code:
.. SPDX-License-Identifier: GPL-2.0

=============================
Network Function Representors
=============================

This document describes the semantics and usage of representor netdevices, as
used to control internal switching on SmartNICs.  For the closely-related port
representors on physical (multi-port) switches, see
:ref:`Documentation/networking/switchdev.rst <switchdev>`.

Motivation
----------

Since the mid-2010s, network cards have started offering more complex
virtualisation capabilities than the legacy SR-IOV approach (with its simple
MAC/VLAN-based switching model) can support.  This led to a desire to offload
software-defined networks (such as OpenVSwitch) to these NICs to specify the
network connectivity of each function.  The resulting designs are variously
called SmartNICs or DPUs.

Network function representors bring the standard Linux networking stack to
virtual switches and IOV devices.  Just as each physical port of a Linux-
controlled switch has a separate netdev, so does each virtual port of a virtual
switch.
When the system boots, and before any offload is configured, all packets from
the virtual functions appear in the networking stack of the PF via the
representors.  The PF can thus always communicate freely with the virtual
functions.
The PF can configure standard Linux forwarding between representors, the uplink
or any other netdev (routing, bridging, TC classifiers).

Thus, a representor is both a control plane object (representing the function in
administrative commands) and a data plane object (one end of a virtual pipe).
As a virtual link endpoint, the representor can be configured like any other
netdevice; in some cases (e.g. link state) the representee will follow the
representor's configuration, while in others there are separate APIs to
configure the representee.

Definitions
-----------

This document uses the term "switchdev function" to refer to the PCIe function
which has administrative control over the virtual switch on the device.
Typically, this will be a PF, but conceivably a NIC could be configured to grant
these administrative privileges instead to a VF or SF (subfunction).
Depending on NIC design, a multi-port NIC might have a single switchdev function
for the whole device or might have a separate virtual switch, and hence
switchdev function, for each physical network port.
If the NIC supports nested switching, there might be separate switchdev
functions for each nested switch, in which case each switchdev function should
only create representors for the ports on the (sub-)switch it directly
administers.

A "representee" is the object that a representor represents.  So for example in
the case of a VF representor, the representee is the corresponding VF.

What does a representor do?
---------------------------

A representor has three main roles.

1. It is used to configure the network connection the representee sees, e.g.
   link up/down, MTU, etc.  For instance, bringing the representor
   administratively UP should cause the representee to see a link up / carrier
   on event.
2. It provides the slow path for traffic which does not hit any offloaded
   fast-path rules in the virtual switch.  Packets transmitted on the
   representor netdevice should be delivered to the representee; packets
   transmitted by the representee which fail to match any switching rule should
   be received on the representor netdevice.  (That is, there is a virtual pipe
   connecting the representor to the representee, similar in concept to a veth
   pair.)
   This allows software switch implementations (such as OpenVSwitch or a Linux
   bridge) to forward packets between representees and the rest of the network.
3. It acts as a handle by which switching rules (such as TC filters) can refer
   to the representee, allowing these rules to be offloaded.

The combination of 2) and 3) means that the behaviour (apart from performance)
should be the same whether a TC filter is offloaded or not.  E.g. a TC rule
on a VF representor applies in software to packets received on that representor
netdevice, while in hardware offload it would apply to packets transmitted by
the representee VF.  Conversely, a mirred egress redirect to a VF representor
corresponds in hardware to delivery directly to the representee VF.

What functions should have a representor?
-----------------------------------------

Essentially, for each virtual port on the device's internal switch, there
should be a representor.
Some vendors have chosen to omit representors for the uplink and the physical
network port, which can simplify usage (the uplink netdev becomes in effect the
physical port's representor) but does not generalise to devices with multiple
ports or uplinks.

Thus, the following should all have representors:

 - VFs belonging to the switchdev function.
 - Other PFs on the local PCIe controller, and any VFs belonging to them.
 - PFs and VFs on external PCIe controllers on the device (e.g. for any embedded
   System-on-Chip within the SmartNIC).
 - PFs and VFs with other personalities, including network block devices (such
   as a vDPA virtio-blk PF backed by remote/distributed storage), if (and only
   if) their network access is implemented through a virtual switch port. [#]_
   Note that such functions can require a representor despite the representee
   not having a netdev.
 - Subfunctions (SFs) belonging to any of the above PFs or VFs, if they have
   their own port on the switch (as opposed to using their parent PF's port).
 - Any accelerators or plugins on the device whose interface to the network is
   through a virtual switch port, even if they do not have a corresponding PCIe
   PF or VF.

This allows the entire switching behaviour of the NIC to be controlled through
representor TC rules.

It is a common misunderstanding to conflate virtual ports with PCIe virtual
functions or their netdevs.  While in simple cases there will be a 1:1
correspondence between VF netdevices and VF representors, more advanced device
configurations may not follow this.
A PCIe function which does not have network access through the internal switch
(not even indirectly through the hardware implementation of whatever services
the function provides) should *not* have a representor (even if it has a
netdev).
Such a function has no switch virtual port for the representor to configure or
to be the other end of the virtual pipe.
The representor represents the virtual port, not the PCIe function nor the 'end
user' netdevice.

.. [#] The concept here is that a hardware IP stack in the device performs the
   translation between block DMA requests and network packets, so that only
   network packets pass through the virtual port onto the switch.  The network
   access that the IP stack "sees" would then be configurable through tc rules;
   e.g. its traffic might all be wrapped in a specific VLAN or VxLAN.  However,
   any needed configuration of the block device *qua* block device, not being a
   networking entity, would not be appropriate for the representor and would
   thus use some other channel such as devlink.
   Contrast this with the case of a virtio-blk implementation which forwards the
   DMA requests unchanged to another PF whose driver then initiates and
   terminates IP traffic in software; in that case the DMA traffic would *not*
   run over the virtual switch and the virtio-blk PF should thus *not* have a
   representor.

How are representors created?
-----------------------------

The driver instance attached to the switchdev function should, for each virtual
port on the switch, create a pure-software netdevice which has some form of
in-kernel reference to the switchdev function's own netdevice or driver private
data (``netdev_priv()``).
This may be by enumerating ports at probe time, reacting dynamically to the
creation and destruction of ports at run time, or a combination of the two.

The operations of the representor netdevice will generally involve acting
through the switchdev function.  For example, ``ndo_start_xmit()`` might send
the packet through a hardware TX queue attached to the switchdev function, with
either packet metadata or queue configuration marking it for delivery to the
representee.

How are representors identified?
--------------------------------

The representor netdevice should *not* directly refer to a PCIe device (e.g.
through ``net_dev->dev.parent`` / ``SET_NETDEV_DEV()``), either of the
representee or of the switchdev function.
Instead, it should implement the ``ndo_get_devlink_port()`` netdevice op, which
the kernel uses to provide the ``phys_switch_id`` and ``phys_port_name`` sysfs
nodes.  (Some legacy drivers implement ``ndo_get_port_parent_id()`` and
``ndo_get_phys_port_name()`` directly, but this is deprecated.)  See
:ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>` for the
details of this API.

It is expected that userland will use this information (e.g. through udev rules)
to construct an appropriately informative name or alias for the netdevice.  For
instance if the switchdev function is ``eth4`` then a representor with a
``phys_port_name`` of ``p0pf1vf2`` might be renamed ``eth4pf1vf2rep``.

There are as yet no established conventions for naming representors which do not
correspond to PCIe functions (e.g. accelerators and plugins).

How do representors interact with TC rules?
-------------------------------------------

Any TC rule on a representor applies (in software TC) to packets received by
that representor netdevice.  Thus, if the delivery part of the rule corresponds
to another port on the virtual switch, the driver may choose to offload it to
hardware, applying it to packets transmitted by the representee.

Similarly, since a TC mirred egress action targeting the representor would (in
software) send the packet through the representor (and thus indirectly deliver
it to the representee), hardware offload should interpret this as delivery to
the representee.

As a simple example, if ``PORT_DEV`` is the physical port representor and
``REP_DEV`` is a VF representor, the following rules::

    tc filter add dev $REP_DEV parent ffff: protocol ipv4 flower \
        action mirred egress redirect dev $PORT_DEV
    tc filter add dev $PORT_DEV parent ffff: protocol ipv4 flower skip_sw \
        action mirred egress mirror dev $REP_DEV

would mean that all IPv4 packets from the VF are sent out the physical port, and
all IPv4 packets received on the physical port are delivered to the VF in
addition to ``PORT_DEV``.  (Note that without ``skip_sw`` on the second rule,
the VF would get two copies, as the packet reception on ``PORT_DEV`` would
trigger the TC rule again and mirror the packet to ``REP_DEV``.)

On devices without separate port and uplink representors, ``PORT_DEV`` would
instead be the switchdev function's own uplink netdevice.

Of course the rules can (if supported by the NIC) include packet-modifying
actions (e.g. VLAN push/pop), which should be performed by the virtual switch.

Tunnel encapsulation and decapsulation are rather more complicated, as they
involve a third netdevice (a tunnel netdev operating in metadata mode, such as
a VxLAN device created with ``ip link add vxlan0 type vxlan external``) and
require an IP address to be bound to the underlay device (e.g. switchdev
function uplink netdev or port representor).  TC rules such as::

    tc filter add dev $REP_DEV parent ffff: flower \
        action tunnel_key set id $VNI src_ip $LOCAL_IP dst_ip $REMOTE_IP \
                              dst_port 4789 \
        action mirred egress redirect dev vxlan0
    tc filter add dev vxlan0 parent ffff: flower enc_src_ip $REMOTE_IP \
        enc_dst_ip $LOCAL_IP enc_key_id $VNI enc_dst_port 4789 \
        action tunnel_key unset action mirred egress redirect dev $REP_DEV

where ``LOCAL_IP`` is an IP address bound to ``PORT_DEV``, and ``REMOTE_IP`` is
another IP address on the same subnet, mean that packets sent by the VF should
be VxLAN encapsulated and sent out the physical port (the driver has to deduce
this by a route lookup of ``LOCAL_IP`` leading to ``PORT_DEV``, and also
perform an ARP/neighbour table lookup to find the MAC addresses to use in the
outer Ethernet frame), while UDP packets received on the physical port with UDP
port 4789 should be parsed as VxLAN and, if their VSID matches ``$VNI``,
decapsulated and forwarded to the VF.

If this all seems complicated, just remember the 'golden rule' of TC offload:
the hardware should ensure the same final results as if the packets were
processed through the slow path, traversed software TC (except ignoring any
``skip_hw`` rules and applying any ``skip_sw`` rules) and were transmitted or
received through the representor netdevices.

Configuring the representee's MAC
---------------------------------

The representee's link state is controlled through the representor.  Setting the
representor administratively UP or DOWN should cause carrier ON or OFF at the
representee.

Setting an MTU on the representor should cause that same MTU to be reported to
the representee.
(On hardware that allows configuring separate and distinct MTU and MRU values,
the representor MTU should correspond to the representee's MRU and vice-versa.)

Currently there is no way to use the representor to set the station permanent
MAC address of the representee; other methods available to do this include:

 - legacy SR-IOV (``ip link set DEVICE vf NUM mac LLADDR``)
 - devlink port function (see **devlink-port(8)** and
   :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`)


code:
.. SPDX-License-Identifier: GPL-2.0

======================
RxRPC Network Protocol
======================

The RxRPC protocol driver provides a reliable two-phase transport on top of UDP
that can be used to perform RxRPC remote operations.  This is done over sockets
of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and
receive data, aborts and errors.

Contents of this document:

 (#) Overview.

 (#) RxRPC protocol summary.

 (#) AF_RXRPC driver model.

 (#) Control messages.

 (#) Socket options.

 (#) Security.

 (#) Example client usage.

 (#) Example server usage.

 (#) AF_RXRPC kernel interface.

 (#) Configurable parameters.

Overview
========

RxRPC is a two-layer protocol.  There is a session layer which provides
reliable virtual connections using UDP over IPv4 (or IPv6) as the transport
layer, but implements a real network protocol; and there's the presentation
layer which renders structured data to binary blobs and back again using XDR
(as does SunRPC)::

		+-------------+
		| Application |
		+-------------+
		|     XDR     |		Presentation
		+-------------+
		|    RxRPC    |		Session
		+-------------+
		|     UDP     |		Transport
		+-------------+

AF_RXRPC provides:

 (1) Part of an RxRPC facility for both kernel and userspace applications by
     making the session part of it a Linux network protocol (AF_RXRPC).

 (2) A two-phase protocol.  The client transmits a blob (the request) and then
     receives a blob (the reply), and the server receives the request and then
     transmits the reply.

 (3) Retention of the reusable bits of the transport system set up for one call
     to speed up subsequent calls.

 (4) A secure protocol, using the Linux kernel's key retention facility to
     manage security on the client end.  The server end must of necessity be
     more active in security negotiations.

AF_RXRPC does not provide XDR marshalling/presentation facilities.  That is
left to the application.  AF_RXRPC only deals in blobs.  Even the operation ID
is just the first four bytes of the request blob, and as such is beyond the
kernel's interest.

Sockets of AF_RXRPC family are:

 (1) created as type SOCK_DGRAM;

 (2) provided with a protocol of the type of underlying transport they're going
     to use - currently only PF_INET is supported.

The Andrew File System (AFS) is an example of an application that uses this and
that has both kernel (filesystem) and userspace (utility) components.

RxRPC Protocol Summary
======================

An overview of the RxRPC protocol:

 (#) RxRPC sits on top of another networking protocol (UDP is the only option
     currently), and uses this to provide network transport.  UDP ports, for
     example, provide transport endpoints.

 (#) RxRPC supports multiple virtual "connections" from any given transport
     endpoint, thus allowing the endpoints to be shared, even to the same
     remote endpoint.

 (#) Each connection goes to a particular "service".  A connection may not go
     to multiple services.  A service may be considered the RxRPC equivalent of
     a port number.  AF_RXRPC permits multiple services to share an endpoint.

 (#) Client-originating packets are marked, thus a transport endpoint can be
     shared between client and server connections (connections have a
     direction).

 (#) Up to a billion connections may be supported concurrently between one
     local transport endpoint and one service on one remote endpoint.  An RxRPC
     connection is described by seven numbers::

	Local address	}
	Local port	} Transport (UDP) address
	Remote address	}
	Remote port	}
	Direction
	Connection ID
	Service ID

 (#) Each RxRPC operation is a "call".  A connection may make up to four
     billion calls, but only up to four calls may be in progress on a
     connection at any one time.

 (#) Calls are two-phase and asymmetric: the client sends its request data,
     which the service receives; then the service transmits the reply data
     which the client receives.

 (#) The data blobs are of indefinite size, the end of a phase is marked with a
     flag in the packet.  The number of packets of data making up one blob may
     not exceed 4 billion, however, as this would cause the sequence number to
     wrap.

 (#) The first four bytes of the request data are the service operation ID.

 (#) Security is negotiated on a per-connection basis.  The connection is
     initiated by the first data packet on it arriving.  If security is
     requested, the server then issues a "challenge" and then the client
     replies with a "response".  If the response is successful, the security is
     set for the lifetime of that connection, and all subsequent calls made
     upon it use that same security.  In the event that the server lets a
     connection lapse before the client, the security will be renegotiated if
     the client uses the connection again.

 (#) Calls use ACK packets to handle reliability.  Data packets are also
     explicitly sequenced per call.

 (#) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs.
     A hard-ACK indicates to the far side that all the data received to a point
     has been received and processed; a soft-ACK indicates that the data has
     been received but may yet be discarded and re-requested.  The sender may
     not discard any transmittable packets until they've been hard-ACK'd.

 (#) Reception of a reply data packet implicitly hard-ACK's all the data
     packets that make up the request.

 (#) An call is complete when the request has been sent, the reply has been
     received and the final hard-ACK on the last packet of the reply has
     reached the server.

 (#) An call may be aborted by either end at any time up to its completion.

AF_RXRPC Driver Model
=====================

About the AF_RXRPC driver:

 (#) The AF_RXRPC protocol transparently uses internal sockets of the transport
     protocol to represent transport endpoints.

 (#) AF_RXRPC sockets map onto RxRPC connection bundles.  Actual RxRPC
     connections are handled transparently.  One client socket may be used to
     make multiple simultaneous calls to the same service.  One server socket
     may handle calls from many clients.

 (#) Additional parallel client connections will be initiated to support extra
     concurrent calls, up to a tunable limit.

 (#) Each connection is retained for a certain amount of time [tunable] after
     the last call currently using it has completed in case a new call is made
     that could reuse it.

 (#) Each internal UDP socket is retained [tunable] for a certain amount of
     time [tunable] after the last connection using it discarded, in case a new
     connection is made that could use it.

 (#) A client-side connection is only shared between calls if they have
     the same key struct describing their security (and assuming the calls
     would otherwise share the connection).  Non-secured calls would also be
     able to share connections with each other.

 (#) A server-side connection is shared if the client says it is.

 (#) ACK'ing is handled by the protocol driver automatically, including ping
     replying.

 (#) SO_KEEPALIVE automatically pings the other side to keep the connection
     alive [TODO].

 (#) If an ICMP error is received, all calls affected by that error will be
     aborted with an appropriate network error passed through recvmsg().

Interaction with the user of the RxRPC socket:

 (#) A socket is made into a server socket by binding an address with a
     non-zero service ID.

 (#) In the client, sending a request is achieved with one or more sendmsgs,
     followed by the reply being received with one or more recvmsgs.

 (#) The first sendmsg for a request to be sent from a client contains a tag to
     be used in all other sendmsgs or recvmsgs associated with that call.  The
     tag is carried in the control data.

 (#) connect() is used to supply a default destination address for a client
     socket.  This may be overridden by supplying an alternate address to the
     first sendmsg() of a call (struct msghdr::msg_name).

 (#) If connect() is called on an unbound client, a random local port will
     bound before the operation takes place.

 (#) A server socket may also be used to make client calls.  To do this, the
     first sendmsg() of the call must specify the target address.  The server's
     transport endpoint is used to send the packets.

 (#) Once the application has received the last message associated with a call,
     the tag is guaranteed not to be seen again, and so it can be used to pin
     client resources.  A new call can then be initiated with the same tag
     without fear of interference.

 (#) In the server, a request is received with one or more recvmsgs, then the
     the reply is transmitted with one or more sendmsgs, and then the final ACK
     is received with a last recvmsg.

 (#) When sending data for a call, sendmsg is given MSG_MORE if there's more
     data to come on that call.

 (#) When receiving data for a call, recvmsg flags MSG_MORE if there's more
     data to come for that call.

 (#) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg
     to indicate the terminal message for that call.

 (#) A call may be aborted by adding an abort control message to the control
     data.  Issuing an abort terminates the kernel's use of that call's tag.
     Any messages waiting in the receive queue for that call will be discarded.

 (#) Aborts, busy notifications and challenge packets are delivered by recvmsg,
     and control data messages will be set to indicate the context.  Receiving
     an abort or a busy message terminates the kernel's use of that call's tag.

 (#) The control data part of the msghdr struct is used for a number of things:

     (#) The tag of the intended or affected call.

     (#) Sending or receiving errors, aborts and busy notifications.

     (#) Notifications of incoming calls.

     (#) Sending debug requests and receiving debug replies [TODO].

 (#) When the kernel has received and set up an incoming call, it sends a
     message to server application to let it know there's a new call awaiting
     its acceptance [recvmsg reports a special control message].  The server
     application then uses sendmsg to assign a tag to the new call.  Once that
     is done, the first part of the request data will be delivered by recvmsg.

 (#) The server application has to provide the server socket with a keyring of
     secret keys corresponding to the security types it permits.  When a secure
     connection is being set up, the kernel looks up the appropriate secret key
     in the keyring and then sends a challenge packet to the client and
     receives a response packet.  The kernel then checks the authorisation of
     the packet and either aborts the connection or sets up the security.

 (#) The name of the key a client will use to secure its communications is
     nominated by a socket option.

Notes on sendmsg:

 (#) MSG_WAITALL can be set to tell sendmsg to ignore signals if the peer is
     making progress at accepting packets within a reasonable time such that we
     manage to queue up all the data for transmission.  This requires the
     client to accept at least one packet per 2*RTT time period.

     If this isn't set, sendmsg() will return immediately, either returning
     EINTR/ERESTARTSYS if nothing was consumed or returning the amount of data
     consumed.

Notes on recvmsg:

 (#) If there's a sequence of data messages belonging to a particular call on
     the receive queue, then recvmsg will keep working through them until:

     (a) it meets the end of that call's received data,

     (b) it meets a non-data message,

     (c) it meets a message belonging to a different call, or

     (d) it fills the user buffer.

     If recvmsg is called in blocking mode, it will keep sleeping, awaiting the
     reception of further data, until one of the above four conditions is met.

 (2) MSG_PEEK operates similarly, but will return immediately if it has put any
     data in the buffer rather than sleeping until it can fill the buffer.

 (3) If a data message is only partially consumed in filling a user buffer,
     then the remainder of that message will be left on the front of the queue
     for the next taker.  MSG_TRUNC will never be flagged.

 (4) If there is more data to be had on a call (it hasn't copied the last byte
     of the last data message in that phase yet), then MSG_MORE will be
     flagged.

Control Messages
================

AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex
calls, to invoke certain actions and to report certain conditions.  These are:

	=======================	=== ===========	===============================
	MESSAGE ID		SRT DATA	MEANING
	=======================	=== ===========	===============================
	RXRPC_USER_CALL_ID	sr- User ID	App's call specifier
	RXRPC_ABORT		srt Abort code	Abort code to issue/received
	RXRPC_ACK		-rt n/a		Final ACK received
	RXRPC_NET_ERROR		-rt error num	Network error on call
	RXRPC_BUSY		-rt n/a		Call rejected (server busy)
	RXRPC_LOCAL_ERROR	-rt error num	Local error encountered
	RXRPC_NEW_CALL		-r- n/a		New call received
	RXRPC_ACCEPT		s-- n/a		Accept new call
	RXRPC_EXCLUSIVE_CALL	s-- n/a		Make an exclusive client call
	RXRPC_UPGRADE_SERVICE	s-- n/a		Client call can be upgraded
	RXRPC_TX_LENGTH		s-- data len	Total length of Tx data
	=======================	=== ===========	===============================

	(SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message)

 (#) RXRPC_USER_CALL_ID

     This is used to indicate the application's call ID.  It's an unsigned long
     that the app specifies in the client by attaching it to the first data
     message or in the server by passing it in association with an RXRPC_ACCEPT
     message.  recvmsg() passes it in conjunction with all messages except
     those of the RXRPC_NEW_CALL message.

 (#) RXRPC_ABORT

     This is can be used by an application to abort a call by passing it to
     sendmsg, or it can be delivered by recvmsg to indicate a remote abort was
     received.  Either way, it must be associated with an RXRPC_USER_CALL_ID to
     specify the call affected.  If an abort is being sent, then error EBADSLT
     will be returned if there is no call with that user ID.

 (#) RXRPC_ACK

     This is delivered to a server application to indicate that the final ACK
     of a call was received from the client.  It will be associated with an
     RXRPC_USER_CALL_ID to indicate the call that's now complete.

 (#) RXRPC_NET_ERROR

     This is delivered to an application to indicate that an ICMP error message
     was encountered in the process of trying to talk to the peer.  An
     errno-class integer value will be included in the control message data
     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
     affected.

 (#) RXRPC_BUSY

     This is delivered to a client application to indicate that a call was
     rejected by the server due to the server being busy.  It will be
     associated with an RXRPC_USER_CALL_ID to indicate the rejected call.

 (#) RXRPC_LOCAL_ERROR

     This is delivered to an application to indicate that a local error was
     encountered and that a call has been aborted because of it.  An
     errno-class integer value will be included in the control message data
     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
     affected.

 (#) RXRPC_NEW_CALL

     This is delivered to indicate to a server application that a new call has
     arrived and is awaiting acceptance.  No user ID is associated with this,
     as a user ID must subsequently be assigned by doing an RXRPC_ACCEPT.

 (#) RXRPC_ACCEPT

     This is used by a server application to attempt to accept a call and
     assign it a user ID.  It should be associated with an RXRPC_USER_CALL_ID
     to indicate the user ID to be assigned.  If there is no call to be
     accepted (it may have timed out, been aborted, etc.), then sendmsg will
     return error ENODATA.  If the user ID is already in use by another call,
     then error EBADSLT will be returned.

 (#) RXRPC_EXCLUSIVE_CALL

     This is used to indicate that a client call should be made on a one-off
     connection.  The connection is discarded once the call has terminated.

 (#) RXRPC_UPGRADE_SERVICE

     This is used to make a client call to probe if the specified service ID
     may be upgraded by the server.  The caller must check msg_name returned to
     recvmsg() for the service ID actually in use.  The operation probed must
     be one that takes the same arguments in both services.

     Once this has been used to establish the upgrade capability (or lack
     thereof) of the server, the service ID returned should be used for all
     future communication to that server and RXRPC_UPGRADE_SERVICE should no
     longer be set.

 (#) RXRPC_TX_LENGTH

     This is used to inform the kernel of the total amount of data that is
     going to be transmitted by a call (whether in a client request or a
     service response).  If given, it allows the kernel to encrypt from the
     userspace buffer directly to the packet buffers, rather than copying into
     the buffer and then encrypting in place.  This may only be given with the
     first sendmsg() providing data for a call.  EMSGSIZE will be generated if
     the amount of data actually given is different.

     This takes a parameter of __s64 type that indicates how much will be
     transmitted.  This may not be less than zero.

The symbol RXRPC__SUPPORTED is defined as one more than the highest control
message type supported.  At run time this can be queried by means of the
RXRPC_SUPPORTED_CMSG socket option (see below).

==============
SOCKET OPTIONS
==============

AF_RXRPC sockets support a few socket options at the SOL_RXRPC level:

 (#) RXRPC_SECURITY_KEY

     This is used to specify the description of the key to be used.  The key is
     extracted from the calling process's keyrings with request_key() and
     should be of "rxrpc" type.

     The optval pointer points to the description string, and optlen indicates
     how long the string is, without the NUL terminator.

 (#) RXRPC_SECURITY_KEYRING

     Similar to above but specifies a keyring of server secret keys to use (key
     type "keyring").  See the "Security" section.

 (#) RXRPC_EXCLUSIVE_CONNECTION

     This is used to request that new connections should be used for each call
     made subsequently on this socket.  optval should be NULL and optlen 0.

 (#) RXRPC_MIN_SECURITY_LEVEL

     This is used to specify the minimum security level required for calls on
     this socket.  optval must point to an int containing one of the following
     values:

     (a) RXRPC_SECURITY_PLAIN

	 Encrypted checksum only.

     (b) RXRPC_SECURITY_AUTH

	 Encrypted checksum plus packet padded and first eight bytes of packet
	 encrypted - which includes the actual packet length.

     (c) RXRPC_SECURITY_ENCRYPT

	 Encrypted checksum plus entire packet padded and encrypted, including
	 actual packet length.

 (#) RXRPC_UPGRADEABLE_SERVICE

     This is used to indicate that a service socket with two bindings may
     upgrade one bound service to the other if requested by the client.  optval
     must point to an array of two unsigned short ints.  The first is the
     service ID to upgrade from and the second the service ID to upgrade to.

 (#) RXRPC_SUPPORTED_CMSG

     This is a read-only option that writes an int into the buffer indicating
     the highest control message type supported.

========
SECURITY
========

Currently, only the kerberos 4 equivalent protocol has been implemented
(security index 2 - rxkad).  This requires the rxkad module to be loaded and,
on the client, tickets of the appropriate type to be obtained from the AFS
kaserver or the kerberos server and installed as "rxrpc" type keys.  This is
normally done using the klog program.  An example simple klog program can be
found at:

	http://people.redhat.com/~dhowells/rxrpc/klog.c

The payload provided to add_key() on the client should be of the following
form::

	struct rxrpc_key_sec2_v1 {
		uint16_t	security_index;	/* 2 */
		uint16_t	ticket_length;	/* length of ticket[] */
		uint32_t	expiry;		/* time at which expires */
		uint8_t		kvno;		/* key version number */
		uint8_t		__pad[3];
		uint8_t		session_key[8];	/* DES session key */
		uint8_t		ticket[0];	/* the encrypted ticket */
	};

Where the ticket blob is just appended to the above structure.

For the server, keys of type "rxrpc_s" must be made available to the server.
They have a description of "<serviceID>:<securityIndex>" (eg: "52:2" for an
rxkad key for the AFS VL service).  When such a key is created, it should be
given the server's secret key as the instantiation data (see the example
below).

	add_key("rxrpc_s", "52:2", secret_key, 8, keyring);

A keyring is passed to the server socket by naming it in a sockopt.  The server
socket then looks the server secret keys up in this keyring when secure
incoming connections are made.  This can be seen in an example program that can
be found at:

	http://people.redhat.com/~dhowells/rxrpc/listen.c

====================
EXAMPLE CLIENT USAGE
====================

A client would issue an operation by:

 (1) An RxRPC socket is set up by::

	client = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);

     Where the third parameter indicates the protocol family of the transport
     socket used - usually IPv4 but it can also be IPv6 [TODO].

 (2) A local address can optionally be bound::

	struct sockaddr_rxrpc srx = {
		.srx_family	= AF_RXRPC,
		.srx_service	= 0,  /* we're a client */
		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
		.transport.sin_family	= AF_INET,
		.transport.sin_port	= htons(7000), /* AFS callback */
		.transport.sin_address	= 0,  /* all local interfaces */
	};
	bind(client, &srx, sizeof(srx));

     This specifies the local UDP port to be used.  If not given, a random
     non-privileged port will be used.  A UDP port may be shared between
     several unrelated RxRPC sockets.  Security is handled on a basis of
     per-RxRPC virtual connection.

 (3) The security is set::

	const char *key = "AFS:cambridge.redhat.com";
	setsockopt(client, SOL_RXRPC, RXRPC_SECURITY_KEY, key, strlen(key));

     This issues a request_key() to get the key representing the security
     context.  The minimum security level can be set::

	unsigned int sec = RXRPC_SECURITY_ENCRYPT;
	setsockopt(client, SOL_RXRPC, RXRPC_MIN_SECURITY_LEVEL,
		   &sec, sizeof(sec));

 (4) The server to be contacted can then be specified (alternatively this can
     be done through sendmsg)::

	struct sockaddr_rxrpc srx = {
		.srx_family	= AF_RXRPC,
		.srx_service	= VL_SERVICE_ID,
		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
		.transport.sin_family	= AF_INET,
		.transport.sin_port	= htons(7005), /* AFS volume manager */
		.transport.sin_address	= ...,
	};
	connect(client, &srx, sizeof(srx));

 (5) The request data should then be posted to the server socket using a series
     of sendmsg() calls, each with the following control message attached:

	==================	===================================
	RXRPC_USER_CALL_ID	specifies the user ID for this call
	==================	===================================

     MSG_MORE should be set in msghdr::msg_flags on all but the last part of
     the request.  Multiple requests may be made simultaneously.

     An RXRPC_TX_LENGTH control message can also be specified on the first
     sendmsg() call.

     If a call is intended to go to a destination other than the default
     specified through connect(), then msghdr::msg_name should be set on the
     first request message of that call.

 (6) The reply data will then be posted to the server socket for recvmsg() to
     pick up.  MSG_MORE will be flagged by recvmsg() if there's more reply data
     for a particular call to be read.  MSG_EOR will be set on the terminal
     read for a call.

     All data will be delivered with the following control message attached:

	RXRPC_USER_CALL_ID	- specifies the user ID for this call

     If an abort or error occurred, this will be returned in the control data
     buffer instead, and MSG_EOR will be flagged to indicate the end of that
     call.

A client may ask for a service ID it knows and ask that this be upgraded to a
better service if one is available by supplying RXRPC_UPGRADE_SERVICE on the
first sendmsg() of a call.  The client should then check srx_service in the
msg_name filled in by recvmsg() when collecting the result.  srx_service will
hold the same value as given to sendmsg() if the upgrade request was ignored by
the service - otherwise it will be altered to indicate the service ID the
server upgraded to.  Note that the upgraded service ID is chosen by the server.
The caller has to wait until it sees the service ID in the reply before sending
any more calls (further calls to the same destination will be blocked until the
probe is concluded).

Example Server Usage
====================

A server would be set up to accept operations in the following manner:

 (1) An RxRPC socket is created by::

	server = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);

     Where the third parameter indicates the address type of the transport
     socket used - usually IPv4.

 (2) Security is set up if desired by giving the socket a keyring with server
     secret keys in it::

	keyring = add_key("keyring", "AFSkeys", NULL, 0,
			  KEY_SPEC_PROCESS_KEYRING);

	const char secret_key[8] = {
		0xa7, 0x83, 0x8a, 0xcb, 0xc7, 0x83, 0xec, 0x94 };
	add_key("rxrpc_s", "52:2", secret_key, 8, keyring);

	setsockopt(server, SOL_RXRPC, RXRPC_SECURITY_KEYRING, "AFSkeys", 7);

     The keyring can be manipulated after it has been given to the socket. This
     permits the server to add more keys, replace keys, etc. while it is live.

 (3) A local address must then be bound::

	struct sockaddr_rxrpc srx = {
		.srx_family	= AF_RXRPC,
		.srx_service	= VL_SERVICE_ID, /* RxRPC service ID */
		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
		.transport.sin_family	= AF_INET,
		.transport.sin_port	= htons(7000), /* AFS callback */
		.transport.sin_address	= 0,  /* all local interfaces */
	};
	bind(server, &srx, sizeof(srx));

     More than one service ID may be bound to a socket, provided the transport
     parameters are the same.  The limit is currently two.  To do this, bind()
     should be called twice.

 (4) If service upgrading is required, first two service IDs must have been
     bound and then the following option must be set::

	unsigned short service_ids[2] = { from_ID, to_ID };
	setsockopt(server, SOL_RXRPC, RXRPC_UPGRADEABLE_SERVICE,
		   service_ids, sizeof(service_ids));

     This will automatically upgrade connections on service from_ID to service
     to_ID if they request it.  This will be reflected in msg_name obtained
     through recvmsg() when the request data is delivered to userspace.

 (5) The server is then set to listen out for incoming calls::

	listen(server, 100);

 (6) The kernel notifies the server of pending incoming connections by sending
     it a message for each.  This is received with recvmsg() on the server
     socket.  It has no data, and has a single dataless control message
     attached::

	RXRPC_NEW_CALL

     The address that can be passed back by recvmsg() at this point should be
     ignored since the call for which the message was posted may have gone by
     the time it is accepted - in which case the first call still on the queue
     will be accepted.

 (7) The server then accepts the new call by issuing a sendmsg() with two
     pieces of control data and no actual data:

	==================	==============================
	RXRPC_ACCEPT		indicate connection acceptance
	RXRPC_USER_CALL_ID	specify user ID for this call
	==================	==============================

 (8) The first request data packet will then be posted to the server socket for
     recvmsg() to pick up.  At that point, the RxRPC address for the call can
     be read from the address fields in the msghdr struct.

     Subsequent request data will be posted to the server socket for recvmsg()
     to collect as it arrives.  All but the last piece of the request data will
     be delivered with MSG_MORE flagged.

     All data will be delivered with the following control message attached:

	==================	===================================
	RXRPC_USER_CALL_ID	specifies the user ID for this call
	==================	===================================

 (9) The reply data should then be posted to the server socket using a series
     of sendmsg() calls, each with the following control messages attached:

	==================	===================================
	RXRPC_USER_CALL_ID	specifies the user ID for this call
	==================	===================================

     MSG_MORE should be set in msghdr::msg_flags on all but the last message
     for a particular call.

(10) The final ACK from the client will be posted for retrieval by recvmsg()
     when it is received.  It will take the form of a dataless message with two
     control messages attached:

	==================	===================================
	RXRPC_USER_CALL_ID	specifies the user ID for this call
	RXRPC_ACK		indicates final ACK (no data)
	==================	===================================

     MSG_EOR will be flagged to indicate that this is the final message for
     this call.

(11) Up to the point the final packet of reply data is sent, the call can be
     aborted by calling sendmsg() with a dataless message with the following
     control messages attached:

	==================	===================================
	RXRPC_USER_CALL_ID	specifies the user ID for this call
	RXRPC_ABORT		indicates abort code (4 byte data)
	==================	===================================

     Any packets waiting in the socket's receive queue will be discarded if
     this is issued.

Note that all the communications for a particular service take place through
the one server socket, using control messages on sendmsg() and recvmsg() to
determine the call affected.

AF_RXRPC Kernel Interface
=========================

The AF_RXRPC module also provides an interface for use by in-kernel utilities
such as the AFS filesystem.  This permits such a utility to:

 (1) Use different keys directly on individual client calls on one socket
     rather than having to open a whole slew of sockets, one for each key it
     might want to use.

 (2) Avoid having RxRPC call request_key() at the point of issue of a call or
     opening of a socket.  Instead the utility is responsible for requesting a
     key at the appropriate point.  AFS, for instance, would do this during VFS
     operations such as open() or unlink().  The key is then handed through
     when the call is initiated.

 (3) Request the use of something other than GFP_KERNEL to allocate memory.

 (4) Avoid the overhead of using the recvmsg() call.  RxRPC messages can be
     intercepted before they get put into the socket Rx queue and the socket
     buffers manipulated directly.

To use the RxRPC facility, a kernel utility must still open an AF_RXRPC socket,
bind an address as appropriate and listen if it's to be a server socket, but
then it passes this to the kernel interface functions.

The kernel interface functions are as follows:

 (#) Begin a new client call::

	struct rxrpc_call *
	rxrpc_kernel_begin_call(struct socket *sock,
				struct sockaddr_rxrpc *srx,
				struct key *key,
				unsigned long user_call_ID,
				s64 tx_total_len,
				gfp_t gfp,
				rxrpc_notify_rx_t notify_rx,
				bool upgrade,
				bool intr,
				unsigned int debug_id);

     This allocates the infrastructure to make a new RxRPC call and assigns
     call and connection numbers.  The call will be made on the UDP port that
     the socket is bound to.  The call will go to the destination address of a
     connected client socket unless an alternative is supplied (srx is
     non-NULL).

     If a key is supplied then this will be used to secure the call instead of
     the key bound to the socket with the RXRPC_SECURITY_KEY sockopt.  Calls
     secured in this way will still share connections if at all possible.

     The user_call_ID is equivalent to that supplied to sendmsg() in the
     control data buffer.  It is entirely feasible to use this to point to a
     kernel data structure.

     tx_total_len is the amount of data the caller is intending to transmit
     with this call (or -1 if unknown at this point).  Setting the data size
     allows the kernel to encrypt directly to the packet buffers, thereby
     saving a copy.  The value may not be less than -1.

     notify_rx is a pointer to a function to be called when events such as
     incoming data packets or remote aborts happen.

     upgrade should be set to true if a client operation should request that
     the server upgrade the service to a better one.  The resultant service ID
     is returned by rxrpc_kernel_recv_data().

     intr should be set to true if the call should be interruptible.  If this
     is not set, this function may not return until a channel has been
     allocated; if it is set, the function may return -ERESTARTSYS.

     debug_id is the call debugging ID to be used for tracing.  This can be
     obtained by atomically incrementing rxrpc_debug_id.

     If this function is successful, an opaque reference to the RxRPC call is
     returned.  The caller now holds a reference on this and it must be
     properly ended.

 (#) End a client call::

	void rxrpc_kernel_end_call(struct socket *sock,
				   struct rxrpc_call *call);

     This is used to end a previously begun call.  The user_call_ID is expunged
     from AF_RXRPC's knowledge and will not be seen again in association with
     the specified call.

 (#) Send data through a call::

	typedef void (*rxrpc_notify_end_tx_t)(struct sock *sk,
					      unsigned long user_call_ID,
					      struct sk_buff *skb);

	int rxrpc_kernel_send_data(struct socket *sock,
				   struct rxrpc_call *call,
				   struct msghdr *msg,
				   size_t len,
				   rxrpc_notify_end_tx_t notify_end_rx);

     This is used to supply either the request part of a client call or the
     reply part of a server call.  msg.msg_iovlen and msg.msg_iov specify the
     data buffers to be used.  msg_iov may not be NULL and must point
     exclusively to in-kernel virtual addresses.  msg.msg_flags may be given
     MSG_MORE if there will be subsequent data sends for this call.

     The msg must not specify a destination address, control data or any flags
     other than MSG_MORE.  len is the total amount of data to transmit.

     notify_end_rx can be NULL or it can be used to specify a function to be
     called when the call changes state to end the Tx phase.  This function is
     called with the call-state spinlock held to prevent any reply or final ACK
     from being delivered first.

 (#) Receive data from a call::

	int rxrpc_kernel_recv_data(struct socket *sock,
				   struct rxrpc_call *call,
				   void *buf,
				   size_t size,
				   size_t *_offset,
				   bool want_more,
				   u32 *_abort,
				   u16 *_service)

      This is used to receive data from either the reply part of a client call
      or the request part of a service call.  buf and size specify how much
      data is desired and where to store it.  *_offset is added on to buf and
      subtracted from size internally; the amount copied into the buffer is
      added to *_offset before returning.

      want_more should be true if further data will be required after this is
      satisfied and false if this is the last item of the receive phase.

      There are three normal returns: 0 if the buffer was filled and want_more
      was true; 1 if the buffer was filled, the last DATA packet has been
      emptied and want_more was false; and -EAGAIN if the function needs to be
      called again.

      If the last DATA packet is processed but the buffer contains less than
      the amount requested, EBADMSG is returned.  If want_more wasn't set, but
      more data was available, EMSGSIZE is returned.

      If a remote ABORT is detected, the abort code received will be stored in
      ``*_abort`` and ECONNABORTED will be returned.

      The service ID that the call ended up with is returned into *_service.
      This can be used to see if a call got a service upgrade.

 (#) Abort a call??

     ::

	void rxrpc_kernel_abort_call(struct socket *sock,
				     struct rxrpc_call *call,
				     u32 abort_code);

     This is used to abort a call if it's still in an abortable state.  The
     abort code specified will be placed in the ABORT message sent.

 (#) Intercept received RxRPC messages::

	typedef void (*rxrpc_interceptor_t)(struct sock *sk,
					    unsigned long user_call_ID,
					    struct sk_buff *skb);

	void
	rxrpc_kernel_intercept_rx_messages(struct socket *sock,
					   rxrpc_interceptor_t interceptor);

     This installs an interceptor function on the specified AF_RXRPC socket.
     All messages that would otherwise wind up in the socket's Rx queue are
     then diverted to this function.  Note that care must be taken to process
     the messages in the right order to maintain DATA message sequentiality.

     The interceptor function itself is provided with the address of the socket
     and handling the incoming message, the ID assigned by the kernel utility
     to the call and the socket buffer containing the message.

     The skb->mark field indicates the type of message:

	===============================	=======================================
	Mark				Meaning
	===============================	=======================================
	RXRPC_SKB_MARK_DATA		Data message
	RXRPC_SKB_MARK_FINAL_ACK	Final ACK received for an incoming call
	RXRPC_SKB_MARK_BUSY		Client call rejected as server busy
	RXRPC_SKB_MARK_REMOTE_ABORT	Call aborted by peer
	RXRPC_SKB_MARK_NET_ERROR	Network error detected
	RXRPC_SKB_MARK_LOCAL_ERROR	Local error encountered
	RXRPC_SKB_MARK_NEW_CALL		New incoming call awaiting acceptance
	===============================	=======================================

     The remote abort message can be probed with rxrpc_kernel_get_abort_code().
     The two error messages can be probed with rxrpc_kernel_get_error_number().
     A new call can be accepted with rxrpc_kernel_accept_call().

     Data messages can have their contents extracted with the usual bunch of
     socket buffer manipulation functions.  A data message can be determined to
     be the last one in a sequence with rxrpc_kernel_is_data_last().  When a
     data message has been used up, rxrpc_kernel_data_consumed() should be
     called on it.

     Messages should be handled to rxrpc_kernel_free_skb() to dispose of.  It
     is possible to get extra refs on all types of message for later freeing,
     but this may pin the state of a call until the message is finally freed.

 (#) Accept an incoming call::

	struct rxrpc_call *
	rxrpc_kernel_accept_call(struct socket *sock,
				 unsigned long user_call_ID);

     This is used to accept an incoming call and to assign it a call ID.  This
     function is similar to rxrpc_kernel_begin_call() and calls accepted must
     be ended in the same way.

     If this function is successful, an opaque reference to the RxRPC call is
     returned.  The caller now holds a reference on this and it must be
     properly ended.

 (#) Reject an incoming call::

	int rxrpc_kernel_reject_call(struct socket *sock);

     This is used to reject the first incoming call on the socket's queue with
     a BUSY message.  -ENODATA is returned if there were no incoming calls.
     Other errors may be returned if the call had been aborted (-ECONNABORTED)
     or had timed out (-ETIME).

 (#) Allocate a null key for doing anonymous security::

	struct key *rxrpc_get_null_key(const char *keyname);

     This is used to allocate a null RxRPC key that can be used to indicate
     anonymous security for a particular domain.

 (#) Get the peer address of a call::

	void rxrpc_kernel_get_peer(struct socket *sock, struct rxrpc_call *call,
				   struct sockaddr_rxrpc *_srx);

     This is used to find the remote peer address of a call.

 (#) Set the total transmit data size on a call::

	void rxrpc_kernel_set_tx_length(struct socket *sock,
					struct rxrpc_call *call,
					s64 tx_total_len);

     This sets the amount of data that the caller is intending to transmit on a
     call.  It's intended to be used for setting the reply size as the request
     size should be set when the call is begun.  tx_total_len may not be less
     than zero.

 (#) Get call RTT::

	u64 rxrpc_kernel_get_rtt(struct socket *sock, struct rxrpc_call *call);

     Get the RTT time to the peer in use by a call.  The value returned is in
     nanoseconds.

 (#) Check call still alive::

	bool rxrpc_kernel_check_life(struct socket *sock,
				     struct rxrpc_call *call,
				     u32 *_life);
	void rxrpc_kernel_probe_life(struct socket *sock,
				     struct rxrpc_call *call);

     The first function passes back in ``*_life`` a number that is updated when
     ACKs are received from the peer (notably including PING RESPONSE ACKs
     which we can elicit by sending PING ACKs to see if the call still exists
     on the server).  The caller should compare the numbers of two calls to see
     if the call is still alive after waiting for a suitable interval.  It also
     returns true as long as the call hasn't yet reached the completed state.

     This allows the caller to work out if the server is still contactable and
     if the call is still alive on the server while waiting for the server to
     process a client operation.

     The second function causes a ping ACK to be transmitted to try to provoke
     the peer into responding, which would then cause the value returned by the
     first function to change.  Note that this must be called in TASK_RUNNING
     state.

 (#) Get remote client epoch::

	u32 rxrpc_kernel_get_epoch(struct socket *sock,
				   struct rxrpc_call *call)

     This allows the epoch that's contained in packets of an incoming client
     call to be queried.  This value is returned.  The function always
     successful if the call is still in progress.  It shouldn't be called once
     the call has expired.  Note that calling this on a local client call only
     returns the local epoch.

     This value can be used to determine if the remote client has been
     restarted as it shouldn't change otherwise.

 (#) Set the maxmimum lifespan on a call::

	void rxrpc_kernel_set_max_life(struct socket *sock,
				       struct rxrpc_call *call,
				       unsigned long hard_timeout)

     This sets the maximum lifespan on a call to hard_timeout (which is in
     jiffies).  In the event of the timeout occurring, the call will be
     aborted and -ETIME or -ETIMEDOUT will be returned.

 (#) Apply the RXRPC_MIN_SECURITY_LEVEL sockopt to a socket from within in the
     kernel::

       int rxrpc_sock_set_min_security_level(struct sock *sk,
					     unsigned int val);

     This specifies the minimum security level required for calls on this
     socket.

Configurable Parameters
=======================

The RxRPC protocol driver has a number of configurable parameters that can be
adjusted through sysctls in /proc/net/rxrpc/:

 (#) req_ack_delay

     The amount of time in milliseconds after receiving a packet with the
     request-ack flag set before we honour the flag and actually send the
     requested ack.

     Usually the other side won't stop sending packets until the advertised
     reception window is full (to a maximum of 255 packets), so delaying the
     ACK permits several packets to be ACK'd in one go.

 (#) soft_ack_delay

     The amount of time in milliseconds after receiving a new packet before we
     generate a soft-ACK to tell the sender that it doesn't need to resend.

 (#) idle_ack_delay

     The amount of time in milliseconds after all the packets currently in the
     received queue have been consumed before we generate a hard-ACK to tell
     the sender it can free its buffers, assuming no other reason occurs that
     we would send an ACK.

 (#) resend_timeout

     The amount of time in milliseconds after transmitting a packet before we
     transmit it again, assuming no ACK is received from the receiver telling
     us they got it.

 (#) max_call_lifetime

     The maximum amount of time in seconds that a call may be in progress
     before we preemptively kill it.

 (#) dead_call_expiry

     The amount of time in seconds before we remove a dead call from the call
     list.  Dead calls are kept around for a little while for the purpose of
     repeating ACK and ABORT packets.

 (#) connection_expiry

     The amount of time in seconds after a connection was last used before we
     remove it from the connection list.  While a connection is in existence,
     it serves as a placeholder for negotiated security; when it is deleted,
     the security must be renegotiated.

 (#) transport_expiry

     The amount of time in seconds after a transport was last used before we
     remove it from the transport list.  While a transport is in existence, it
     serves to anchor the peer data and keeps the connection ID counter.

 (#) rxrpc_rx_window_size

     The size of the receive window in packets.  This is the maximum number of
     unconsumed received packets we're willing to hold in memory for any
     particular call.

 (#) rxrpc_rx_mtu

     The maximum packet MTU size that we're willing to receive in bytes.  This
     indicates to the peer whether we're willing to accept jumbo packets.

 (#) rxrpc_rx_jumbo_max

     The maximum number of packets that we're willing to accept in a jumbo
     packet.  Non-terminal packets in a jumbo packet must contain a four byte
     header plus exactly 1412 bytes of data.  The terminal packet must contain
     a four byte header plus any amount of data.  In any event, a jumbo packet
     may not exceed rxrpc_rx_mtu in size.

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

=====================================
Scaling in the Linux Networking Stack
=====================================

Introduction
============

This document describes a set of complementary techniques in the Linux
networking stack to increase parallelism and improve performance for
multi-processor systems.

The following technologies are described:

- RSS: Receive Side Scaling
- RPS: Receive Packet Steering
- RFS: Receive Flow Steering
- Accelerated Receive Flow Steering
- XPS: Transmit Packet Steering

RSS: Receive Side Scaling
=========================

Contemporary NICs support multiple receive and transmit descriptor queues
(multi-queue). On reception, a NIC can send different packets to different
queues to distribute processing among CPUs. The NIC distributes packets by
applying a filter to each packet that assigns it to one of a small number
of logical flows. Packets for each flow are steered to a separate receive
queue, which in turn can be processed by separate CPUs. This mechanism is
generally known as “Receive-side Scaling” (RSS). The goal of RSS and
the other scaling techniques is to increase performance uniformly.
Multi-queue distribution can also be used for traffic prioritization, but
that is not the focus of these techniques.

The filter used in RSS is typically a hash function over the network
and/or transport layer headers-- for example, a 4-tuple hash over
IP addresses and TCP ports of a packet. The most common hardware
implementation of RSS uses a 128-entry indirection table where each entry
stores a queue number. The receive queue for a packet is determined
by masking out the low order seven bits of the computed hash for the
packet (usually a Toeplitz hash), taking this number as a key into the
indirection table and reading the corresponding value.

Some advanced NICs allow steering packets to queues based on
programmable filters. For example, webserver bound TCP port 80 packets
can be directed to their own receive queue. Such “n-tuple” filters can
be configured from ethtool (--config-ntuple).

RSS Configuration
-----------------

The driver for a multi-queue capable NIC typically provides a kernel
module parameter for specifying the number of hardware queues to
configure. In the bnx2x driver, for instance, this parameter is called
num_queues. A typical RSS configuration would be to have one receive queue
for each CPU if the device supports enough queues, or otherwise at least
one for each memory domain, where a memory domain is a set of CPUs that
share a particular memory level (L1, L2, NUMA node, etc.).

The indirection table of an RSS device, which resolves a queue by masked
hash, is usually programmed by the driver at initialization. The
default mapping is to distribute the queues evenly in the table, but the
indirection table can be retrieved and modified at runtime using ethtool
commands (--show-rxfh-indir and --set-rxfh-indir). Modifying the
indirection table could be done to give different queues different
relative weights.

RSS IRQ Configuration
~~~~~~~~~~~~~~~~~~~~~

Each receive queue has a separate IRQ associated with it. The NIC triggers
this to notify a CPU when new packets arrive on the given queue. The
signaling path for PCIe devices uses message signaled interrupts (MSI-X),
that can route each interrupt to a particular CPU. The active mapping
of queues to IRQs can be determined from /proc/interrupts. By default,
an IRQ may be handled on any CPU. Because a non-negligible part of packet
processing takes place in receive interrupt handling, it is advantageous
to spread receive interrupts between CPUs. To manually adjust the IRQ
affinity of each interrupt see Documentation/core-api/irq/irq-affinity.rst. Some systems
will be running irqbalance, a daemon that dynamically optimizes IRQ
assignments and as a result may override any manual settings.

Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~

RSS should be enabled when latency is a concern or whenever receive
interrupt processing forms a bottleneck. Spreading load between CPUs
decreases queue length. For low latency networking, the optimal setting
is to allocate as many queues as there are CPUs in the system (or the
NIC maximum, if lower). The most efficient high-rate configuration
is likely the one with the smallest number of receive queues where no
receive queue overflows due to a saturated CPU, because in default
mode with interrupt coalescing enabled, the aggregate number of
interrupts (and thus work) grows with each additional queue.

Per-cpu load can be observed using the mpstat utility, but note that on
processors with hyperthreading (HT), each hyperthread is represented as
a separate CPU. For interrupt handling, HT has shown no benefit in
initial tests, so limit the number of queues to the number of CPU cores
in the system.

RPS: Receive Packet Steering
============================

Receive Packet Steering (RPS) is logically a software implementation of
RSS. Being in software, it is necessarily called later in the datapath.
Whereas RSS selects the queue and hence CPU that will run the hardware
interrupt handler, RPS selects the CPU to perform protocol processing
above the interrupt handler. This is accomplished by placing the packet
on the desired CPU’s backlog queue and waking up the CPU for processing.
RPS has some advantages over RSS:

1) it can be used with any NIC
2) software filters can easily be added to hash over new protocols
3) it does not increase hardware device interrupt rate (although it does
   introduce inter-processor interrupts (IPIs))

RPS is called during bottom half of the receive interrupt handler, when
a driver sends a packet up the network stack with netif_rx() or
netif_receive_skb(). These call the get_rps_cpu() function, which
selects the queue that should process a packet.

The first step in determining the target CPU for RPS is to calculate a
flow hash over the packet’s addresses or ports (2-tuple or 4-tuple hash
depending on the protocol). This serves as a consistent hash of the
associated flow of the packet. The hash is either provided by hardware
or will be computed in the stack. Capable hardware can pass the hash in
the receive descriptor for the packet; this would usually be the same
hash used for RSS (e.g. computed Toeplitz hash). The hash is saved in
skb->hash and can be used elsewhere in the stack as a hash of the
packet’s flow.

Each receive hardware queue has an associated list of CPUs to which
RPS may enqueue packets for processing. For each received packet,
an index into the list is computed from the flow hash modulo the size
of the list. The indexed CPU is the target for processing the packet,
and the packet is queued to the tail of that CPU’s backlog queue. At
the end of the bottom half routine, IPIs are sent to any CPUs for which
packets have been queued to their backlog queue. The IPI wakes backlog
processing on the remote CPU, and any queued packets are then processed
up the networking stack.

RPS Configuration
-----------------

RPS requires a kernel compiled with the CONFIG_RPS kconfig symbol (on
by default for SMP). Even when compiled in, RPS remains disabled until
explicitly configured. The list of CPUs to which RPS may forward traffic
can be configured for each receive queue using a sysfs file entry::

  /sys/class/net/<dev>/queues/rx-<n>/rps_cpus

This file implements a bitmap of CPUs. RPS is disabled when it is zero
(the default), in which case packets are processed on the interrupting
CPU. Documentation/core-api/irq/irq-affinity.rst explains how CPUs are assigned to
the bitmap.

Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~

For a single queue device, a typical RPS configuration would be to set
the rps_cpus to the CPUs in the same memory domain of the interrupting
CPU. If NUMA locality is not an issue, this could also be all CPUs in
the system. At high interrupt rate, it might be wise to exclude the
interrupting CPU from the map since that already performs much work.

For a multi-queue system, if RSS is configured so that a hardware
receive queue is mapped to each CPU, then RPS is probably redundant
and unnecessary. If there are fewer hardware queues than CPUs, then
RPS might be beneficial if the rps_cpus for each queue are the ones that
share the same memory domain as the interrupting CPU for that queue.

RPS Flow Limit
--------------

RPS scales kernel receive processing across CPUs without introducing
reordering. The trade-off to sending all packets from the same flow
to the same CPU is CPU load imbalance if flows vary in packet rate.
In the extreme case a single flow dominates traffic. Especially on
common server workloads with many concurrent connections, such
behavior indicates a problem such as a misconfiguration or spoofed
source Denial of Service attack.

Flow Limit is an optional RPS feature that prioritizes small flows
during CPU contention by dropping packets from large flows slightly
ahead of those from small flows. It is active only when an RPS or RFS
destination CPU approaches saturation.  Once a CPU's input packet
queue exceeds half the maximum queue length (as set by sysctl
net.core.netdev_max_backlog), the kernel starts a per-flow packet
count over the last 256 packets. If a flow exceeds a set ratio (by
default, half) of these packets when a new packet arrives, then the
new packet is dropped. Packets from other flows are still only
dropped once the input packet queue reaches netdev_max_backlog.
No packets are dropped when the input packet queue length is below
the threshold, so flow limit does not sever connections outright:
even large flows maintain connectivity.

Interface
~~~~~~~~~

Flow limit is compiled in by default (CONFIG_NET_FLOW_LIMIT), but not
turned on. It is implemented for each CPU independently (to avoid lock
and cache contention) and toggled per CPU by setting the relevant bit
in sysctl net.core.flow_limit_cpu_bitmap. It exposes the same CPU
bitmap interface as rps_cpus (see above) when called from procfs::

  /proc/sys/net/core/flow_limit_cpu_bitmap

Per-flow rate is calculated by hashing each packet into a hashtable
bucket and incrementing a per-bucket counter. The hash function is
the same that selects a CPU in RPS, but as the number of buckets can
be much larger than the number of CPUs, flow limit has finer-grained
identification of large flows and fewer false positives. The default
table has 4096 buckets. This value can be modified through sysctl::

  net.core.flow_limit_table_len

The value is only consulted when a new table is allocated. Modifying
it does not update active tables.

Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~

Flow limit is useful on systems with many concurrent connections,
where a single connection taking up 50% of a CPU indicates a problem.
In such environments, enable the feature on all CPUs that handle
network rx interrupts (as set in /proc/irq/N/smp_affinity).

The feature depends on the input packet queue length to exceed
the flow limit threshold (50%) + the flow history length (256).
Setting net.core.netdev_max_backlog to either 1000 or 10000
performed well in experiments.

RFS: Receive Flow Steering
==========================

While RPS steers packets solely based on hash, and thus generally
provides good load distribution, it does not take into account
application locality. This is accomplished by Receive Flow Steering
(RFS). The goal of RFS is to increase datacache hitrate by steering
kernel processing of packets to the CPU where the application thread
consuming the packet is running. RFS relies on the same RPS mechanisms
to enqueue packets onto the backlog of another CPU and to wake up that
CPU.

In RFS, packets are not forwarded directly by the value of their hash,
but the hash is used as index into a flow lookup table. This table maps
flows to the CPUs where those flows are being processed. The flow hash
(see RPS section above) is used to calculate the index into this table.
The CPU recorded in each entry is the one which last processed the flow.
If an entry does not hold a valid CPU, then packets mapped to that entry
are steered using plain RPS. Multiple table entries may point to the
same CPU. Indeed, with many flows and few CPUs, it is very likely that
a single application thread handles flows with many different flow hashes.

rps_sock_flow_table is a global flow table that contains the *desired* CPU
for flows: the CPU that is currently processing the flow in userspace.
Each table value is a CPU index that is updated during calls to recvmsg
and sendmsg (specifically, inet_recvmsg(), inet_sendmsg(), inet_sendpage()
and tcp_splice_read()).

When the scheduler moves a thread to a new CPU while it has outstanding
receive packets on the old CPU, packets may arrive out of order. To
avoid this, RFS uses a second flow table to track outstanding packets
for each flow: rps_dev_flow_table is a table specific to each hardware
receive queue of each device. Each table value stores a CPU index and a
counter. The CPU index represents the *current* CPU onto which packets
for this flow are enqueued for further kernel processing. Ideally, kernel
and userspace processing occur on the same CPU, and hence the CPU index
in both tables is identical. This is likely false if the scheduler has
recently migrated a userspace thread while the kernel still has packets
enqueued for kernel processing on the old CPU.

The counter in rps_dev_flow_table values records the length of the current
CPU's backlog when a packet in this flow was last enqueued. Each backlog
queue has a head counter that is incremented on dequeue. A tail counter
is computed as head counter + queue length. In other words, the counter
in rps_dev_flow[i] records the last element in flow i that has
been enqueued onto the currently designated CPU for flow i (of course,
entry i is actually selected by hash and multiple flows may hash to the
same entry i).

And now the trick for avoiding out of order packets: when selecting the
CPU for packet processing (from get_rps_cpu()) the rps_sock_flow table
and the rps_dev_flow table of the queue that the packet was received on
are compared. If the desired CPU for the flow (found in the
rps_sock_flow table) matches the current CPU (found in the rps_dev_flow
table), the packet is enqueued onto that CPU’s backlog. If they differ,
the current CPU is updated to match the desired CPU if one of the
following is true:

  - The current CPU's queue head counter >= the recorded tail counter
    value in rps_dev_flow[i]
  - The current CPU is unset (>= nr_cpu_ids)
  - The current CPU is offline

After this check, the packet is sent to the (possibly updated) current
CPU. These rules aim to ensure that a flow only moves to a new CPU when
there are no packets outstanding on the old CPU, as the outstanding
packets could arrive later than those about to be processed on the new
CPU.

RFS Configuration
-----------------

RFS is only available if the kconfig symbol CONFIG_RPS is enabled (on
by default for SMP). The functionality remains disabled until explicitly
configured. The number of entries in the global flow table is set through::

  /proc/sys/net/core/rps_sock_flow_entries

The number of entries in the per-queue flow table are set through::

  /sys/class/net/<dev>/queues/rx-<n>/rps_flow_cnt

Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~

Both of these need to be set before RFS is enabled for a receive queue.
Values for both are rounded up to the nearest power of two. The
suggested flow count depends on the expected number of active connections
at any given time, which may be significantly less than the number of open
connections. We have found that a value of 32768 for rps_sock_flow_entries
works fairly well on a moderately loaded server.

For a single queue device, the rps_flow_cnt value for the single queue
would normally be configured to the same value as rps_sock_flow_entries.
For a multi-queue device, the rps_flow_cnt for each queue might be
configured as rps_sock_flow_entries / N, where N is the number of
queues. So for instance, if rps_sock_flow_entries is set to 32768 and there
are 16 configured receive queues, rps_flow_cnt for each queue might be
configured as 2048.

Accelerated RFS
===============

Accelerated RFS is to RFS what RSS is to RPS: a hardware-accelerated load
balancing mechanism that uses soft state to steer flows based on where
the application thread consuming the packets of each flow is running.
Accelerated RFS should perform better than RFS since packets are sent
directly to a CPU local to the thread consuming the data. The target CPU
will either be the same CPU where the application runs, or at least a CPU
which is local to the application thread’s CPU in the cache hierarchy.

To enable accelerated RFS, the networking stack calls the
ndo_rx_flow_steer driver function to communicate the desired hardware
queue for packets matching a particular flow. The network stack
automatically calls this function every time a flow entry in
rps_dev_flow_table is updated. The driver in turn uses a device specific
method to program the NIC to steer the packets.

The hardware queue for a flow is derived from the CPU recorded in
rps_dev_flow_table. The stack consults a CPU to hardware queue map which
is maintained by the NIC driver. This is an auto-generated reverse map of
the IRQ affinity table shown by /proc/interrupts. Drivers can use
functions in the cpu_rmap (“CPU affinity reverse map”) kernel library
to populate the map. For each CPU, the corresponding queue in the map is
set to be one whose processing CPU is closest in cache locality.

Accelerated RFS Configuration
-----------------------------

Accelerated RFS is only available if the kernel is compiled with
CONFIG_RFS_ACCEL and support is provided by the NIC device and driver.
It also requires that ntuple filtering is enabled via ethtool. The map
of CPU to queues is automatically deduced from the IRQ affinities
configured for each receive queue by the driver, so no additional
configuration should be necessary.

Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~

This technique should be enabled whenever one wants to use RFS and the
NIC supports hardware acceleration.

XPS: Transmit Packet Steering
=============================

Transmit Packet Steering is a mechanism for intelligently selecting
which transmit queue to use when transmitting a packet on a multi-queue
device. This can be accomplished by recording two kinds of maps, either
a mapping of CPU to hardware queue(s) or a mapping of receive queue(s)
to hardware transmit queue(s).

1. XPS using CPUs map

The goal of this mapping is usually to assign queues
exclusively to a subset of CPUs, where the transmit completions for
these queues are processed on a CPU within this set. This choice
provides two benefits. First, contention on the device queue lock is
significantly reduced since fewer CPUs contend for the same queue
(contention can be eliminated completely if each CPU has its own
transmit queue). Secondly, cache miss rate on transmit completion is
reduced, in particular for data cache lines that hold the sk_buff
structures.

2. XPS using receive queues map

This mapping is used to pick transmit queue based on the receive
queue(s) map configuration set by the administrator. A set of receive
queues can be mapped to a set of transmit queues (many:many), although
the common use case is a 1:1 mapping. This will enable sending packets
on the same queue associations for transmit and receive. This is useful for
busy polling multi-threaded workloads where there are challenges in
associating a given CPU to a given application thread. The application
threads are not pinned to CPUs and each thread handles packets
received on a single queue. The receive queue number is cached in the
socket for the connection. In this model, sending the packets on the same
transmit queue corresponding to the associated receive queue has benefits
in keeping the CPU overhead low. Transmit completion work is locked into
the same queue-association that a given application is polling on. This
avoids the overhead of triggering an interrupt on another CPU. When the
application cleans up the packets during the busy poll, transmit completion
may be processed along with it in the same thread context and so result in
reduced latency.

XPS is configured per transmit queue by setting a bitmap of
CPUs/receive-queues that may use that queue to transmit. The reverse
mapping, from CPUs to transmit queues or from receive-queues to transmit
queues, is computed and maintained for each network device. When
transmitting the first packet in a flow, the function get_xps_queue() is
called to select a queue. This function uses the ID of the receive queue
for the socket connection for a match in the receive queue-to-transmit queue
lookup table. Alternatively, this function can also use the ID of the
running CPU as a key into the CPU-to-queue lookup table. If the
ID matches a single queue, that is used for transmission. If multiple
queues match, one is selected by using the flow hash to compute an index
into the set. When selecting the transmit queue based on receive queue(s)
map, the transmit device is not validated against the receive device as it
requires expensive lookup operation in the datapath.

The queue chosen for transmitting a particular flow is saved in the
corresponding socket structure for the flow (e.g. a TCP connection).
This transmit queue is used for subsequent packets sent on the flow to
prevent out of order (ooo) packets. The choice also amortizes the cost
of calling get_xps_queues() over all packets in the flow. To avoid
ooo packets, the queue for a flow can subsequently only be changed if
skb->ooo_okay is set for a packet in the flow. This flag indicates that
there are no outstanding packets in the flow, so the transmit queue can
change without the risk of generating out of order packets. The
transport layer is responsible for setting ooo_okay appropriately. TCP,
for instance, sets the flag when all data for a connection has been
acknowledged.

XPS Configuration
-----------------

XPS is only available if the kconfig symbol CONFIG_XPS is enabled (on by
default for SMP). If compiled in, it is driver dependent whether, and
how, XPS is configured at device init. The mapping of CPUs/receive-queues
to transmit queue can be inspected and configured using sysfs:

For selection based on CPUs map::

  /sys/class/net/<dev>/queues/tx-<n>/xps_cpus

For selection based on receive-queues map::

  /sys/class/net/<dev>/queues/tx-<n>/xps_rxqs

Suggested Configuration
~~~~~~~~~~~~~~~~~~~~~~~

For a network device with a single transmission queue, XPS configuration
has no effect, since there is no choice in this case. In a multi-queue
system, XPS is preferably configured so that each CPU maps onto one queue.
If there are as many queues as there are CPUs in the system, then each
queue can also map onto one CPU, resulting in exclusive pairings that
experience no contention. If there are fewer queues than CPUs, then the
best CPUs to share a given queue are probably those that share the cache
with the CPU that processes transmit completions for that queue
(transmit interrupts).

For transmit queue selection based on receive queue(s), XPS has to be
explicitly configured mapping receive-queue(s) to transmit queue(s). If the
user configuration for receive-queue map does not apply, then the transmit
queue is selected based on the CPUs map.

Per TX Queue rate limitation
============================

These are rate-limitation mechanisms implemented by HW, where currently
a max-rate attribute is supported, by setting a Mbps value to::

  /sys/class/net/<dev>/queues/tx-<n>/tx_maxrate

A value of zero means disabled, and this is the default.

Further Information
===================
RPS and RFS were introduced in kernel 2.6.35. XPS was incorporated into
2.6.38. Original patches were submitted by Tom Herbert
(therbert@google.com)

Accelerated RFS was introduced in 2.6.35. Original patches were
submitted by Ben Hutchings (bwh@kernel.org)

Authors:

- Tom Herbert (therbert@google.com)
- Willem de Bruijn (willemb@google.com)


code:
.. SPDX-License-Identifier: GPL-2.0

=================
Linux Kernel SCTP
=================

This is the current BETA release of the Linux Kernel SCTP reference
implementation.

SCTP (Stream Control Transmission Protocol) is a IP based, message oriented,
reliable transport protocol, with congestion control, support for
transparent multi-homing, and multiple ordered streams of messages.
RFC2960 defines the core protocol.  The IETF SIGTRAN working group originally
developed the SCTP protocol and later handed the protocol over to the
Transport Area (TSVWG) working group for the continued evolvement of SCTP as a
general purpose transport.

See the IETF website (http://www.ietf.org) for further documents on SCTP.
See http://www.ietf.org/rfc/rfc2960.txt

The initial project goal is to create an Linux kernel reference implementation
of SCTP that is RFC 2960 compliant and provides an programming interface
referred to as the  UDP-style API of the Sockets Extensions for SCTP, as
proposed in IETF Internet-Drafts.

Caveats
=======

- lksctp can be built as statically or as a module.  However, be aware that
  module removal of lksctp is not yet a safe activity.

- There is tentative support for IPv6, but most work has gone towards
  implementation and testing lksctp on IPv4.

For more information, please visit the lksctp project website:

   http://www.sf.net/projects/lksctp

Or contact the lksctp developers through the mailing list:

   <linux-sctp@vger.kernel.org>


code:
.. SPDX-License-Identifier: GPL-2.0

=================
LSM/SeLinux secid
=================

flowi structure:

The secid member in the flow structure is used in LSMs (e.g. SELinux) to indicate
the label of the flow. This label of the flow is currently used in selecting
matching labeled xfrm(s).

If this is an outbound flow, the label is derived from the socket, if any, or
the incoming packet this flow is being generated as a response to (e.g. tcp
resets, timewait ack, etc.). It is also conceivable that the label could be
derived from other sources such as process context, device, etc., in special
cases, as may be appropriate.

If this is an inbound flow, the label is derived from the IPSec security
associations, if any, used by the packet.


code:
.. SPDX-License-Identifier: GPL-2.0

====================
Seg6 Sysfs variables
====================

/proc/sys/net/conf/<iface>/seg6_* variables:
============================================

seg6_enabled - BOOL
	Accept or drop SR-enabled IPv6 packets on this interface.

	Relevant packets are those with SRH present and DA = local.

	* 0 - disabled (default)
	* not 0 - enabled

seg6_require_hmac - INTEGER
	Define HMAC policy for ingress SR-enabled packets on this interface.

	* -1 - Ignore HMAC field
	* 0 - Accept SR packets without HMAC, validate SR packets with HMAC
	* 1 - Drop SR packets without HMAC, validate SR packets with HMAC

	Default is 0.

seg6_flowlabel - INTEGER
	Controls the behaviour of computing the flowlabel of outer
	IPv6 header in case of SR T.encaps

	 == =======================================================
	 -1  set flowlabel to zero.
	  0  copy flowlabel from Inner packet in case of Inner IPv6
	     (Set flowlabel to 0 in case IPv4/L2)
	  1  Compute the flowlabel using seg6_make_flowlabel()
	 == =======================================================

	Default is 0.


code:
.. SPDX-License-Identifier: GPL-2.0

=====================
Segmentation Offloads
=====================

Introduction
============

This document describes a set of techniques in the Linux networking stack
to take advantage of segmentation offload capabilities of various NICs.

The following technologies are described:
 * TCP Segmentation Offload - TSO
 * UDP Fragmentation Offload - UFO
 * IPIP, SIT, GRE, and UDP Tunnel Offloads
 * Generic Segmentation Offload - GSO
 * Generic Receive Offload - GRO
 * Partial Generic Segmentation Offload - GSO_PARTIAL
 * SCTP acceleration with GSO - GSO_BY_FRAGS

TCP Segmentation Offload
========================

TCP segmentation allows a device to segment a single frame into multiple
frames with a data payload size specified in skb_shinfo()->gso_size.
When TCP segmentation requested the bit for either SKB_GSO_TCPV4 or
SKB_GSO_TCPV6 should be set in skb_shinfo()->gso_type and
skb_shinfo()->gso_size should be set to a non-zero value.

TCP segmentation is dependent on support for the use of partial checksum
offload.  For this reason TSO is normally disabled if the Tx checksum
offload for a given device is disabled.

In order to support TCP segmentation offload it is necessary to populate
the network and transport header offsets of the skbuff so that the device
drivers will be able determine the offsets of the IP or IPv6 header and the
TCP header.  In addition as CHECKSUM_PARTIAL is required csum_start should
also point to the TCP header of the packet.

For IPv4 segmentation we support one of two types in terms of the IP ID.
The default behavior is to increment the IP ID with every segment.  If the
GSO type SKB_GSO_TCP_FIXEDID is specified then we will not increment the IP
ID and all segments will use the same IP ID.  If a device has
NETIF_F_TSO_MANGLEID set then the IP ID can be ignored when performing TSO
and we will either increment the IP ID for all frames, or leave it at a
static value based on driver preference.

UDP Fragmentation Offload
=========================

UDP fragmentation offload allows a device to fragment an oversized UDP
datagram into multiple IPv4 fragments.  Many of the requirements for UDP
fragmentation offload are the same as TSO.  However the IPv4 ID for
fragments should not increment as a single IPv4 datagram is fragmented.

UFO is deprecated: modern kernels will no longer generate UFO skbs, but can
still receive them from tuntap and similar devices. Offload of UDP-based
tunnel protocols is still supported.

IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads
========================================================

In addition to the offloads described above it is possible for a frame to
contain additional headers such as an outer tunnel.  In order to account
for such instances an additional set of segmentation offload types were
introduced including SKB_GSO_IPXIP4, SKB_GSO_IPXIP6, SKB_GSO_GRE, and
SKB_GSO_UDP_TUNNEL.  These extra segmentation types are used to identify
cases where there are more than just 1 set of headers.  For example in the
case of IPIP and SIT we should have the network and transport headers moved
from the standard list of headers to "inner" header offsets.

Currently only two levels of headers are supported.  The convention is to
refer to the tunnel headers as the outer headers, while the encapsulated
data is normally referred to as the inner headers.  Below is the list of
calls to access the given headers:

IPIP/SIT Tunnel::

             Outer                  Inner
  MAC        skb_mac_header
  Network    skb_network_header     skb_inner_network_header
  Transport  skb_transport_header

UDP/GRE Tunnel::

             Outer                  Inner
  MAC        skb_mac_header         skb_inner_mac_header
  Network    skb_network_header     skb_inner_network_header
  Transport  skb_transport_header   skb_inner_transport_header

In addition to the above tunnel types there are also SKB_GSO_GRE_CSUM and
SKB_GSO_UDP_TUNNEL_CSUM.  These two additional tunnel types reflect the
fact that the outer header also requests to have a non-zero checksum
included in the outer header.

Finally there is SKB_GSO_TUNNEL_REMCSUM which indicates that a given tunnel
header has requested a remote checksum offload.  In this case the inner
headers will be left with a partial checksum and only the outer header
checksum will be computed.

Generic Segmentation Offload
============================

Generic segmentation offload is a pure software offload that is meant to
deal with cases where device drivers cannot perform the offloads described
above.  What occurs in GSO is that a given skbuff will have its data broken
out over multiple skbuffs that have been resized to match the MSS provided
via skb_shinfo()->gso_size.

Before enabling any hardware segmentation offload a corresponding software
offload is required in GSO.  Otherwise it becomes possible for a frame to
be re-routed between devices and end up being unable to be transmitted.

Generic Receive Offload
=======================

Generic receive offload is the complement to GSO.  Ideally any frame
assembled by GRO should be segmented to create an identical sequence of
frames using GSO, and any sequence of frames segmented by GSO should be
able to be reassembled back to the original by GRO.  The only exception to
this is IPv4 ID in the case that the DF bit is set for a given IP header.
If the value of the IPv4 ID is not sequentially incrementing it will be
altered so that it is when a frame assembled via GRO is segmented via GSO.

Partial Generic Segmentation Offload
====================================

Partial generic segmentation offload is a hybrid between TSO and GSO.  What
it effectively does is take advantage of certain traits of TCP and tunnels
so that instead of having to rewrite the packet headers for each segment
only the inner-most transport header and possibly the outer-most network
header need to be updated.  This allows devices that do not support tunnel
offloads or tunnel offloads with checksum to still make use of segmentation.

With the partial offload what occurs is that all headers excluding the
inner transport header are updated such that they will contain the correct
values for if the header was simply duplicated.  The one exception to this
is the outer IPv4 ID field.  It is up to the device drivers to guarantee
that the IPv4 ID field is incremented in the case that a given header does
not have the DF bit set.

SCTP acceleration with GSO
===========================

SCTP - despite the lack of hardware support - can still take advantage of
GSO to pass one large packet through the network stack, rather than
multiple small packets.

This requires a different approach to other offloads, as SCTP packets
cannot be just segmented to (P)MTU. Rather, the chunks must be contained in
IP segments, padding respected. So unlike regular GSO, SCTP can't just
generate a big skb, set gso_size to the fragmentation point and deliver it
to IP layer.

Instead, the SCTP protocol layer builds an skb with the segments correctly
padded and stored as chained skbs, and skb_segment() splits based on those.
To signal this, gso_size is set to the special value GSO_BY_FRAGS.

Therefore, any code in the core networking stack must be aware of the
possibility that gso_size will be GSO_BY_FRAGS and handle that case
appropriately.

There are some helpers to make this easier:

- skb_is_gso(skb) && skb_is_gso_sctp(skb) is the best way to see if
  an skb is an SCTP GSO skb.

- For size checks, the skb_gso_validate_*_len family of helpers correctly
  considers GSO_BY_FRAGS.

- For manipulating packets, skb_increase_gso_size and skb_decrease_gso_size
  will check for GSO_BY_FRAGS and WARN if asked to manipulate these skbs.

This also affects drivers with the NETIF_F_FRAGLIST & NETIF_F_GSO_SCTP bits
set. Note also that NETIF_F_GSO_SCTP is included in NETIF_F_GSO_SOFTWARE.