code:
.. SPDX-License-Identifier: GPL-2.0

=========================================
How to use packet injection with mac80211
=========================================

mac80211 now allows arbitrary packets to be injected down any Monitor Mode
interface from userland.  The packet you inject needs to be composed in the
following format::

 [ radiotap header  ]
 [ ieee80211 header ]
 [ payload ]

The radiotap format is discussed in
./Documentation/networking/radiotap-headers.rst.

Despite many radiotap parameters being currently defined, most only make sense
to appear on received packets.  The following information is parsed from the
radiotap headers and used to control injection:

 * IEEE80211_RADIOTAP_FLAGS

   =========================  ===========================================
   IEEE80211_RADIOTAP_F_FCS   FCS will be removed and recalculated
   IEEE80211_RADIOTAP_F_WEP   frame will be encrypted if key available
   IEEE80211_RADIOTAP_F_FRAG  frame will be fragmented if longer than the
			      current fragmentation threshold.
   =========================  ===========================================

 * IEEE80211_RADIOTAP_TX_FLAGS

   =============================  ========================================
   IEEE80211_RADIOTAP_F_TX_NOACK  frame should be sent without waiting for
				  an ACK even if it is a unicast frame
   =============================  ========================================

 * IEEE80211_RADIOTAP_RATE

   legacy rate for the transmission (only for devices without own rate control)

 * IEEE80211_RADIOTAP_MCS

   HT rate for the transmission (only for devices without own rate control).
   Also some flags are parsed

   ============================  ========================
   IEEE80211_RADIOTAP_MCS_SGI    use short guard interval
   IEEE80211_RADIOTAP_MCS_BW_40  send in HT40 mode
   ============================  ========================

 * IEEE80211_RADIOTAP_DATA_RETRIES

   number of retries when either IEEE80211_RADIOTAP_RATE or
   IEEE80211_RADIOTAP_MCS was used

 * IEEE80211_RADIOTAP_VHT

   VHT mcs and number of streams used in the transmission (only for devices
   without own rate control). Also other fields are parsed

   flags field
	IEEE80211_RADIOTAP_VHT_FLAG_SGI: use short guard interval

   bandwidth field
	* 1: send using 40MHz channel width
	* 4: send using 80MHz channel width
	* 11: send using 160MHz channel width

The injection code can also skip all other currently defined radiotap fields
facilitating replay of captured radiotap headers directly.

Here is an example valid radiotap header defining some parameters::

	0x00, 0x00, // <-- radiotap version
	0x0b, 0x00, // <- radiotap header length
	0x04, 0x0c, 0x00, 0x00, // <-- bitmap
	0x6c, // <-- rate
	0x0c, //<-- tx power
	0x01 //<-- antenna

The ieee80211 header follows immediately afterwards, looking for example like
this::

	0x08, 0x01, 0x00, 0x00,
	0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
	0x13, 0x22, 0x33, 0x44, 0x55, 0x66,
	0x13, 0x22, 0x33, 0x44, 0x55, 0x66,
	0x10, 0x86

Then lastly there is the payload.

After composing the packet contents, it is sent by send()-ing it to a logical
mac80211 interface that is in Monitor mode.  Libpcap can also be used,
(which is easier than doing the work to bind the socket to the right
interface), along the following lines:::

	ppcap = pcap_open_live(szInterfaceName, 800, 1, 20, szErrbuf);
	...
	r = pcap_inject(ppcap, u8aSendBuffer, nLength);

You can also find a link to a complete inject application here:

https://wireless.wiki.kernel.org/en/users/Documentation/packetspammer

Andy Green <andy@warmcat.com>


code:
interface=wlan0
driver=nl80211

hw_mode=g
channel=1
ssid=mac80211 test

wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
wpa_passphrase=12345678

repo_name: angery/linux


code:
:orphan:

.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

===================================================================
mac80211_hwsim - software simulator of 802.11 radio(s) for mac80211
===================================================================

:Copyright: |copy| 2008, Jouni Malinen <j@w1.fi>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License version 2 as
published by the Free Software Foundation.

Introduction
============

mac80211_hwsim is a Linux kernel module that can be used to simulate
arbitrary number of IEEE 802.11 radios for mac80211. It can be used to
test most of the mac80211 functionality and user space tools (e.g.,
hostapd and wpa_supplicant) in a way that matches very closely with
the normal case of using real WLAN hardware. From the mac80211 view
point, mac80211_hwsim is yet another hardware driver, i.e., no changes
to mac80211 are needed to use this testing tool.

The main goal for mac80211_hwsim is to make it easier for developers
to test their code and work with new features to mac80211, hostapd,
and wpa_supplicant. The simulated radios do not have the limitations
of real hardware, so it is easy to generate an arbitrary test setup
and always reproduce the same setup for future tests. In addition,
since all radio operation is simulated, any channel can be used in
tests regardless of regulatory rules.

mac80211_hwsim kernel module has a parameter 'radios' that can be used
to select how many radios are simulated (default 2). This allows
configuration of both very simply setups (e.g., just a single access
point and a station) or large scale tests (multiple access points with
hundreds of stations).

mac80211_hwsim works by tracking the current channel of each virtual
radio and copying all transmitted frames to all other radios that are
currently enabled and on the same channel as the transmitting
radio. Software encryption in mac80211 is used so that the frames are
actually encrypted over the virtual air interface to allow more
complete testing of encryption.

A global monitoring netdev, hwsim#, is created independent of
mac80211. This interface can be used to monitor all transmitted frames
regardless of channel.

Simple example
==============

This example shows how to use mac80211_hwsim to simulate two radios:
one to act as an access point and the other as a station that
associates with the AP. hostapd and wpa_supplicant are used to take
care of WPA2-PSK authentication. In addition, hostapd is also
processing access point side of association.

::

    # Build mac80211_hwsim as part of kernel configuration

    # Load the module
    modprobe mac80211_hwsim

    # Run hostapd (AP) for wlan0
    hostapd hostapd.conf

    # Run wpa_supplicant (station) for wlan1
    wpa_supplicant -Dnl80211 -iwlan1 -c wpa_supplicant.conf

More test cases are available in hostap.git:
git://w1.fi/srv/git/hostap.git and mac80211_hwsim/tests subdirectory
(http://w1.fi/gitweb/gitweb.cgi?p=hostap.git;a=tree;f=mac80211_hwsim/tests)


code:
ctrl_interface=/var/run/wpa_supplicant

network={
	ssid="mac80211 test"
	psk="12345678"
	key_mgmt=WPA-PSK
	proto=WPA2
	pairwise=CCMP
	group=CCMP
}


code:
.. SPDX-License-Identifier: GPL-2.0

==============================================
Management Component Transport Protocol (MCTP)
==============================================

net/mctp/ contains protocol support for MCTP, as defined by DMTF standard
DSP0236. Physical interface drivers ("bindings" in the specification) are
provided in drivers/net/mctp/.

The core code provides a socket-based interface to send and receive MCTP
messages, through an AF_MCTP, SOCK_DGRAM socket.

Structure: interfaces & networks
================================

The kernel models the local MCTP topology through two items: interfaces and
networks.

An interface (or "link") is an instance of an MCTP physical transport binding
(as defined by DSP0236, section 3.2.47), likely connected to a specific hardware
device. This is represented as a ``struct netdevice``.

A network defines a unique address space for MCTP endpoints by endpoint-ID
(described by DSP0236, section 3.2.31). A network has a user-visible identifier
to allow references from userspace. Route definitions are specific to one
network.

Interfaces are associated with one network. A network may be associated with one
or more interfaces.

If multiple networks are present, each may contain endpoint IDs (EIDs) that are
also present on other networks.

Sockets API
===========

Protocol definitions
--------------------

MCTP uses ``AF_MCTP`` / ``PF_MCTP`` for the address- and protocol- families.
Since MCTP is message-based, only ``SOCK_DGRAM`` sockets are supported.

.. code-block:: C

    int sd = socket(AF_MCTP, SOCK_DGRAM, 0);

The only (current) value for the ``protocol`` argument is 0.

As with all socket address families, source and destination addresses are
specified with a ``sockaddr`` type, with a single-byte endpoint address:

.. code-block:: C

    typedef __u8		mctp_eid_t;

    struct mctp_addr {
            mctp_eid_t		s_addr;
    };

    struct sockaddr_mctp {
            __kernel_sa_family_t smctp_family;
            unsigned int         smctp_network;
            struct mctp_addr     smctp_addr;
            __u8                 smctp_type;
            __u8                 smctp_tag;
    };

    #define MCTP_NET_ANY	0x0
    #define MCTP_ADDR_ANY	0xff

Syscall behaviour
-----------------

The following sections describe the MCTP-specific behaviours of the standard
socket system calls. These behaviours have been chosen to map closely to the
existing sockets APIs.

``bind()`` : set local socket address
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sockets that receive incoming request packets will bind to a local address,
using the ``bind()`` syscall.

.. code-block:: C

    struct sockaddr_mctp addr;

    addr.smctp_family = AF_MCTP;
    addr.smctp_network = MCTP_NET_ANY;
    addr.smctp_addr.s_addr = MCTP_ADDR_ANY;
    addr.smctp_type = MCTP_TYPE_PLDM;
    addr.smctp_tag = MCTP_TAG_OWNER;

    int rc = bind(sd, (struct sockaddr *)&addr, sizeof(addr));

This establishes the local address of the socket. Incoming MCTP messages that
match the network, address, and message type will be received by this socket.
The reference to 'incoming' is important here; a bound socket will only receive
messages with the TO bit set, to indicate an incoming request message, rather
than a response.

The ``smctp_tag`` value will configure the tags accepted from the remote side of
this socket. Given the above, the only valid value is ``MCTP_TAG_OWNER``, which
will result in remotely "owned" tags being routed to this socket. Since
``MCTP_TAG_OWNER`` is set, the 3 least-significant bits of ``smctp_tag`` are not
used; callers must set them to zero.

A ``smctp_network`` value of ``MCTP_NET_ANY`` will configure the socket to
receive incoming packets from any locally-connected network. A specific network
value will cause the socket to only receive incoming messages from that network.

The ``smctp_addr`` field specifies a local address to bind to. A value of
``MCTP_ADDR_ANY`` configures the socket to receive messages addressed to any
local destination EID.

The ``smctp_type`` field specifies which message types to receive. Only the
lower 7 bits of the type is matched on incoming messages (ie., the
most-significant IC bit is not part of the match). This results in the socket
receiving packets with and without a message integrity check footer.

``sendto()``, ``sendmsg()``, ``send()`` : transmit an MCTP message
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

An MCTP message is transmitted using one of the ``sendto()``, ``sendmsg()`` or
``send()`` syscalls. Using ``sendto()`` as the primary example:

.. code-block:: C

    struct sockaddr_mctp addr;
    char buf[14];
    ssize_t len;

    /* set message destination */
    addr.smctp_family = AF_MCTP;
    addr.smctp_network = 0;
    addr.smctp_addr.s_addr = 8;
    addr.smctp_tag = MCTP_TAG_OWNER;
    addr.smctp_type = MCTP_TYPE_ECHO;

    /* arbitrary message to send, with message-type header */
    buf[0] = MCTP_TYPE_ECHO;
    memcpy(buf + 1, "hello, world!", sizeof(buf) - 1);

    len = sendto(sd, buf, sizeof(buf), 0,
                    (struct sockaddr_mctp *)&addr, sizeof(addr));

The network and address fields of ``addr`` define the remote address to send to.
If ``smctp_tag`` has the ``MCTP_TAG_OWNER``, the kernel will ignore any bits set
in ``MCTP_TAG_VALUE``, and generate a tag value suitable for the destination
EID. If ``MCTP_TAG_OWNER`` is not set, the message will be sent with the tag
value as specified. If a tag value cannot be allocated, the system call will
report an errno of ``EAGAIN``.

The application must provide the message type byte as the first byte of the
message buffer passed to ``sendto()``. If a message integrity check is to be
included in the transmitted message, it must also be provided in the message
buffer, and the most-significant bit of the message type byte must be 1.

The ``sendmsg()`` system call allows a more compact argument interface, and the
message buffer to be specified as a scatter-gather list. At present no ancillary
message types (used for the ``msg_control`` data passed to ``sendmsg()``) are
defined.

Transmitting a message on an unconnected socket with ``MCTP_TAG_OWNER``
specified will cause an allocation of a tag, if no valid tag is already
allocated for that destination. The (destination-eid,tag) tuple acts as an
implicit local socket address, to allow the socket to receive responses to this
outgoing message. If any previous allocation has been performed (to for a
different remote EID), that allocation is lost.

Sockets will only receive responses to requests they have sent (with TO=1) and
may only respond (with TO=0) to requests they have received.

``recvfrom()``, ``recvmsg()``, ``recv()`` : receive an MCTP message
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

An MCTP message can be received by an application using one of the
``recvfrom()``, ``recvmsg()``, or ``recv()`` system calls. Using ``recvfrom()``
as the primary example:

.. code-block:: C

    struct sockaddr_mctp addr;
    socklen_t addrlen;
    char buf[14];
    ssize_t len;

    addrlen = sizeof(addr);

    len = recvfrom(sd, buf, sizeof(buf), 0,
                    (struct sockaddr_mctp *)&addr, &addrlen);

    /* We can expect addr to describe an MCTP address */
    assert(addrlen >= sizeof(buf));
    assert(addr.smctp_family == AF_MCTP);

    printf("received %zd bytes from remote EID %d\n", rc, addr.smctp_addr);

The address argument to ``recvfrom`` and ``recvmsg`` is populated with the
remote address of the incoming message, including tag value (this will be needed
in order to reply to the message).

The first byte of the message buffer will contain the message type byte. If an
integrity check follows the message, it will be included in the received buffer.

The ``recv()`` system call behaves in a similar way, but does not provide a
remote address to the application. Therefore, these are only useful if the
remote address is already known, or the message does not require a reply.

Like the send calls, sockets will only receive responses to requests they have
sent (TO=1) and may only respond (TO=0) to requests they have received.

``ioctl(SIOCMCTPALLOCTAG)`` and ``ioctl(SIOCMCTPDROPTAG)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These tags give applications more control over MCTP message tags, by allocating
(and dropping) tag values explicitly, rather than the kernel automatically
allocating a per-message tag at ``sendmsg()`` time.

In general, you will only need to use these ioctls if your MCTP protocol does
not fit the usual request/response model. For example, if you need to persist
tags across multiple requests, or a request may generate more than one response.
In these cases, the ioctls allow you to decouple the tag allocation (and
release) from individual message send and receive operations.

Both ioctls are passed a pointer to a ``struct mctp_ioc_tag_ctl``:

.. code-block:: C

    struct mctp_ioc_tag_ctl {
        mctp_eid_t      peer_addr;
        __u8		tag;
        __u16   	flags;
    };

``SIOCMCTPALLOCTAG`` allocates a tag for a specific peer, which an application
can use in future ``sendmsg()`` calls. The application populates the
``peer_addr`` member with the remote EID. Other fields must be zero.

On return, the ``tag`` member will be populated with the allocated tag value.
The allocated tag will have the following tag bits set:

 - ``MCTP_TAG_OWNER``: it only makes sense to allocate tags if you're the tag
   owner

 - ``MCTP_TAG_PREALLOC``: to indicate to ``sendmsg()`` that this is a
   preallocated tag.

 - ... and the actual tag value, within the least-significant three bits
   (``MCTP_TAG_MASK``). Note that zero is a valid tag value.

The tag value should be used as-is for the ``smctp_tag`` member of ``struct
sockaddr_mctp``.

``SIOCMCTPDROPTAG`` releases a tag that has been previously allocated by a
``SIOCMCTPALLOCTAG`` ioctl. The ``peer_addr`` must be the same as used for the
allocation, and the ``tag`` value must match exactly the tag returned from the
allocation (including the ``MCTP_TAG_OWNER`` and ``MCTP_TAG_PREALLOC`` bits).
The ``flags`` field must be zero.

Kernel internals
================

There are a few possible packet flows in the MCTP stack:

1. local TX to remote endpoint, message <= MTU::

	sendmsg()
	 -> mctp_local_output()
	    : route lookup
	    -> rt->output() (== mctp_route_output)
	       -> dev_queue_xmit()

2. local TX to remote endpoint, message > MTU::

	sendmsg()
	-> mctp_local_output()
	    -> mctp_do_fragment_route()
	       : creates packet-sized skbs. For each new skb:
	       -> rt->output() (== mctp_route_output)
	          -> dev_queue_xmit()

3. remote TX to local endpoint, single-packet message::

	mctp_pkttype_receive()
	: route lookup
	-> rt->output() (== mctp_route_input)
	   : sk_key lookup
	   -> sock_queue_rcv_skb()

4. remote TX to local endpoint, multiple-packet message::

	mctp_pkttype_receive()
	: route lookup
	-> rt->output() (== mctp_route_input)
	   : sk_key lookup
	   : stores skb in struct sk_key->reasm_head

	mctp_pkttype_receive()
	: route lookup
	-> rt->output() (== mctp_route_input)
	   : sk_key lookup
	   : finds existing reassembly in sk_key->reasm_head
	   : appends new fragment
	   -> sock_queue_rcv_skb()

Key refcounts
-------------

 * keys are refed by:

   - a skb: during route output, stored in ``skb->cb``.

   - netns and sock lists.

 * keys can be associated with a device, in which case they hold a
   reference to the dev (set through ``key->dev``, counted through
   ``dev->key_count``). Multiple keys can reference the device.


code:
.. SPDX-License-Identifier: GPL-2.0

====================
MPLS Sysfs variables
====================

/proc/sys/net/mpls/* Variables:
===============================

platform_labels - INTEGER
	Number of entries in the platform label table.  It is not
	possible to configure forwarding for label values equal to or
	greater than the number of platform labels.

	A dense utilization of the entries in the platform label table
	is possible and expected as the platform labels are locally
	allocated.

	If the number of platform label table entries is set to 0 no
	label will be recognized by the kernel and mpls forwarding
	will be disabled.

	Reducing this value will remove all label routing entries that
	no longer fit in the table.

	Possible values: 0 - 1048575

	Default: 0

ip_ttl_propagate - BOOL
	Control whether TTL is propagated from the IPv4/IPv6 header to
	the MPLS header on imposing labels and propagated from the
	MPLS header to the IPv4/IPv6 header on popping the last label.

	If disabled, the MPLS transport network will appear as a
	single hop to transit traffic.

	* 0 - disabled / RFC 3443 [Short] Pipe Model
	* 1 - enabled / RFC 3443 Uniform Model (default)

default_ttl - INTEGER
	Default TTL value to use for MPLS packets where it cannot be
	propagated from an IP header, either because one isn't present
	or ip_ttl_propagate has been disabled.

	Possible values: 1 - 255

	Default: 255

conf/<interface>/input - BOOL
	Control whether packets can be input on this interface.

	If disabled, packets will be discarded without further
	processing.

	* 0 - disabled (default)
	* not 0 - enabled


code:
.. SPDX-License-Identifier: GPL-2.0

=====================
MPTCP Sysfs variables
=====================

/proc/sys/net/mptcp/* Variables
===============================

enabled - BOOLEAN
	Control whether MPTCP sockets can be created.

	MPTCP sockets can be created if the value is 1. This is a
	per-namespace sysctl.

	Default: 1 (enabled)

add_addr_timeout - INTEGER (seconds)
	Set the timeout after which an ADD_ADDR control message will be
	resent to an MPTCP peer that has not acknowledged a previous
	ADD_ADDR message.

	The default value matches TCP_RTO_MAX. This is a per-namespace
	sysctl.

	Default: 120

checksum_enabled - BOOLEAN
	Control whether DSS checksum can be enabled.

	DSS checksum can be enabled if the value is nonzero. This is a
	per-namespace sysctl.

	Default: 0

allow_join_initial_addr_port - BOOLEAN
	Allow peers to send join requests to the IP address and port number used
	by the initial subflow if the value is 1. This controls a flag that is
	sent to the peer at connection time, and whether such join requests are
	accepted or denied.

	Joins to addresses advertised with ADD_ADDR are not affected by this
	value.

	This is a per-namespace sysctl.

	Default: 1

pm_type - INTEGER
	Set the default path manager type to use for each new MPTCP
	socket. In-kernel path management will control subflow
	connections and address advertisements according to
	per-namespace values configured over the MPTCP netlink
	API. Userspace path management puts per-MPTCP-connection subflow
	connection decisions and address advertisements under control of
	a privileged userspace program, at the cost of more netlink
	traffic to propagate all of the related events and commands.

	This is a per-namespace sysctl.

	* 0 - In-kernel path manager
	* 1 - Userspace path manager

	Default: 0

stale_loss_cnt - INTEGER
	The number of MPTCP-level retransmission intervals with no traffic and
	pending outstanding data on a given subflow required to declare it stale.
	The packet scheduler ignores stale subflows.
	A low stale_loss_cnt  value allows for fast active-backup switch-over,
	an high value maximize links utilization on edge scenarios e.g. lossy
	link with high BER or peer pausing the data processing.

	This is a per-namespace sysctl.

	Default: 4


code:
============
MSG_ZEROCOPY
============

Intro
=====

The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.
The feature is currently implemented for TCP and UDP sockets.

Opportunity and Caveats
-----------------------

Copying large buffers between user process and kernel can be
expensive. Linux supports various interfaces that eschew copying,
such as sendpage and splice. The MSG_ZEROCOPY flag extends the
underlying copy avoidance mechanism to common socket send calls.

Copy avoidance is not a free lunch. As implemented, with page pinning,
it replaces per byte copy cost with page accounting and completion
notification overhead. As a result, MSG_ZEROCOPY is generally only
effective at writes over around 10 KB.

Page pinning also changes system call semantics. It temporarily shares
the buffer between process and network stack. Unlike with copying, the
process cannot immediately overwrite the buffer after system call
return without possibly modifying the data in flight. Kernel integrity
is not affected, but a buggy program can possibly corrupt its own data
stream.

The kernel returns a notification when it is safe to modify data.
Converting an existing application to MSG_ZEROCOPY is not always as
trivial as just passing the flag, then.

More Info
---------

Much of this document was derived from a longer paper presented at
netdev 2.1. For more in-depth information see that paper and talk,
the excellent reporting over at LWN.net or read the original code.

  paper, slides, video
    https://netdevconf.org/2.1/session.html?debruijn

  LWN article
    https://lwn.net/Articles/726917/

  patchset
    [PATCH net-next v4 0/9] socket sendmsg MSG_ZEROCOPY
    https://lore.kernel.org/netdev/20170803202945.70750-1-willemdebruijn.kernel@gmail.com

Interface
=========

Passing the MSG_ZEROCOPY flag is the most obvious step to enable copy
avoidance, but not the only one.

Socket Setup
------------

The kernel is permissive when applications pass undefined flags to the
send system call. By default it simply ignores these. To avoid enabling
copy avoidance mode for legacy processes that accidentally already pass
this flag, a process must first signal intent by setting a socket option:

::

	if (setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one)))
		error(1, errno, "setsockopt zerocopy");

Transmission
------------

The change to send (or sendto, sendmsg, sendmmsg) itself is trivial.
Pass the new flag.

::

	ret = send(fd, buf, sizeof(buf), MSG_ZEROCOPY);

A zerocopy failure will return -1 with errno ENOBUFS. This happens if
the socket option was not set, the socket exceeds its optmem limit or
the user exceeds its ulimit on locked pages.

Mixing copy avoidance and copying
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Many workloads have a mixture of large and small buffers. Because copy
avoidance is more expensive than copying for small packets, the
feature is implemented as a flag. It is safe to mix calls with the flag
with those without.

Notifications
-------------

The kernel has to notify the process when it is safe to reuse a
previously passed buffer. It queues completion notifications on the
socket error queue, akin to the transmit timestamping interface.

The notification itself is a simple scalar value. Each socket
maintains an internal unsigned 32-bit counter. Each send call with
MSG_ZEROCOPY that successfully sends data increments the counter. The
counter is not incremented on failure or if called with length zero.
The counter counts system call invocations, not bytes. It wraps after
UINT_MAX calls.

Notification Reception
~~~~~~~~~~~~~~~~~~~~~~

The below snippet demonstrates the API. In the simplest case, each
send syscall is followed by a poll and recvmsg on the error queue.

Reading from the error queue is always a non-blocking operation. The
poll call is there to block until an error is outstanding. It will set
POLLERR in its output flags. That flag does not have to be set in the
events field. Errors are signaled unconditionally.

::

	pfd.fd = fd;
	pfd.events = 0;
	if (poll(&pfd, 1, -1) != 1 || pfd.revents & POLLERR == 0)
		error(1, errno, "poll");

	ret = recvmsg(fd, &msg, MSG_ERRQUEUE);
	if (ret == -1)
		error(1, errno, "recvmsg");

	read_notification(msg);

The example is for demonstration purpose only. In practice, it is more
efficient to not wait for notifications, but read without blocking
every couple of send calls.

Notifications can be processed out of order with other operations on
the socket. A socket that has an error queued would normally block
other operations until the error is read. Zerocopy notifications have
a zero error code, however, to not block send and recv calls.

Notification Batching
~~~~~~~~~~~~~~~~~~~~~

Multiple outstanding packets can be read at once using the recvmmsg
call. This is often not needed. In each message the kernel returns not
a single value, but a range. It coalesces consecutive notifications
while one is outstanding for reception on the error queue.

When a new notification is about to be queued, it checks whether the
new value extends the range of the notification at the tail of the
queue. If so, it drops the new notification packet and instead increases
the range upper value of the outstanding notification.

For protocols that acknowledge data in-order, like TCP, each
notification can be squashed into the previous one, so that no more
than one notification is outstanding at any one point.

Ordered delivery is the common case, but not guaranteed. Notifications
may arrive out of order on retransmission and socket teardown.

Notification Parsing
~~~~~~~~~~~~~~~~~~~~

The below snippet demonstrates how to parse the control message: the
read_notification() call in the previous snippet. A notification
is encoded in the standard error format, sock_extended_err.

The level and type fields in the control data are protocol family
specific, IP_RECVERR or IPV6_RECVERR.

Error origin is the new type SO_EE_ORIGIN_ZEROCOPY. ee_errno is zero,
as explained before, to avoid blocking read and write system calls on
the socket.

The 32-bit notification range is encoded as [ee_info, ee_data]. This
range is inclusive. Other fields in the struct must be treated as
undefined, bar for ee_code, as discussed below.

::

	struct sock_extended_err *serr;
	struct cmsghdr *cm;

	cm = CMSG_FIRSTHDR(msg);
	if (cm->cmsg_level != SOL_IP &&
	    cm->cmsg_type != IP_RECVERR)
		error(1, 0, "cmsg");

	serr = (void *) CMSG_DATA(cm);
	if (serr->ee_errno != 0 ||
	    serr->ee_origin != SO_EE_ORIGIN_ZEROCOPY)
		error(1, 0, "serr");

	printf("completed: %u..%u\n", serr->ee_info, serr->ee_data);

Deferred copies
~~~~~~~~~~~~~~~

Passing flag MSG_ZEROCOPY is a hint to the kernel to apply copy
avoidance, and a contract that the kernel will queue a completion
notification. It is not a guarantee that the copy is elided.

Copy avoidance is not always feasible. Devices that do not support
scatter-gather I/O cannot send packets made up of kernel generated
protocol headers plus zerocopy user data. A packet may need to be
converted to a private copy of data deep in the stack, say to compute
a checksum.

In all these cases, the kernel returns a completion notification when
it releases its hold on the shared pages. That notification may arrive
before the (copied) data is fully transmitted. A zerocopy completion
notification is not a transmit completion notification, therefore.

Deferred copies can be more expensive than a copy immediately in the
system call, if the data is no longer warm in the cache. The process
also incurs notification processing cost for no benefit. For this
reason, the kernel signals if data was completed with a copy, by
setting flag SO_EE_CODE_ZEROCOPY_COPIED in field ee_code on return.
A process may use this signal to stop passing flag MSG_ZEROCOPY on
subsequent requests on the same socket.

Implementation
==============

Loopback
--------

Data sent to local sockets can be queued indefinitely if the receive
process does not read its socket. Unbound notification latency is not
acceptable. For this reason all packets generated with MSG_ZEROCOPY
that are looped to a local socket will incur a deferred copy. This
includes looping onto packet sockets (e.g., tcpdump) and tun devices.

Testing
=======

More realistic example code can be found in the kernel source under
tools/testing/selftests/net/msg_zerocopy.c.

Be cognizant of the loopback constraint. The test can be run between
a pair of hosts. But if run between a local pair of processes, for
instance when run with msg_zerocopy.sh between a veth pair across
namespaces, the test will not show any improvement. For testing, the
loopback restriction can be temporarily relaxed by making
skb_orphan_frags_rx identical to skb_orphan_frags.


code:
.. SPDX-License-Identifier: GPL-2.0

===========================================
HOWTO for multiqueue network device support
===========================================

Section 1: Base driver requirements for implementing multiqueue support
=======================================================================

Intro: Kernel support for multiqueue devices
---------------------------------------------------------

Kernel support for multiqueue devices is always present.

Base drivers are required to use the new alloc_etherdev_mq() or
alloc_netdev_mq() functions to allocate the subqueues for the device.  The
underlying kernel API will take care of the allocation and deallocation of
the subqueue memory, as well as netdev configuration of where the queues
exist in memory.

The base driver will also need to manage the queues as it does the global
netdev->queue_lock today.  Therefore base drivers should use the
netif_{start|stop|wake}_subqueue() functions to manage each queue while the
device is still operational.  netdev->queue_lock is still used when the device
comes online or when it's completely shut down (unregister_netdev(), etc.).

Section 2: Qdisc support for multiqueue devices
===============================================

Currently two qdiscs are optimized for multiqueue devices.  The first is the
default pfifo_fast qdisc.  This qdisc supports one qdisc per hardware queue.
A new round-robin qdisc, sch_multiq also supports multiple hardware queues. The
qdisc is responsible for classifying the skb's and then directing the skb's to
bands and queues based on the value in skb->queue_mapping.  Use this field in
the base driver to determine which queue to send the skb to.

sch_multiq has been added for hardware that wishes to avoid head-of-line
blocking.  It will cycle though the bands and verify that the hardware queue
associated with the band is not stopped prior to dequeuing a packet.

On qdisc load, the number of bands is based on the number of queues on the
hardware.  Once the association is made, any skb with skb->queue_mapping set,
will be queued to the band associated with the hardware queue.

Section 3: Brief howto using MULTIQ for multiqueue devices
==========================================================

The userspace command 'tc,' part of the iproute2 package, is used to configure
qdiscs.  To add the MULTIQ qdisc to your network device, assuming the device
is called eth0, run the following command::

    # tc qdisc add dev eth0 root handle 1: multiq

The qdisc will allocate the number of bands to equal the number of queues that
the device reports, and bring the qdisc online.  Assuming eth0 has 4 Tx
queues, the band mapping would look like::

    band 0 => queue 0
    band 1 => queue 1
    band 2 => queue 2
    band 3 => queue 3

Traffic will begin flowing through each queue based on either the simple_tx_hash
function or based on netdev->select_queue() if you have it defined.

The behavior of tc filters remains the same.  However a new tc action,
skbedit, has been added.  Assuming you wanted to route all traffic to a
specific host, for example 192.168.0.3, through a specific queue you could use
this action and establish a filter such as::

    tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \
	    match ip dst 192.168.0.3 \
	    action skbedit queue_mapping 3

:Author: Alexander Duyck <alexander.h.duyck@intel.com>
:Original Author: Peter P. Waskiewicz Jr. <peter.p.waskiewicz.jr@intel.com>


code:
======================================================
Net DIM - Generic Network Dynamic Interrupt Moderation
======================================================

:Author: Tal Gilboa <talgi@mellanox.com>

.. contents:: :depth: 2

Assumptions
===========

This document assumes the reader has basic knowledge in network drivers
and in general interrupt moderation.

Introduction
============

Dynamic Interrupt Moderation (DIM) (in networking) refers to changing the
interrupt moderation configuration of a channel in order to optimize packet
processing. The mechanism includes an algorithm which decides if and how to
change moderation parameters for a channel, usually by performing an analysis on
runtime data sampled from the system. Net DIM is such a mechanism. In each
iteration of the algorithm, it analyses a given sample of the data, compares it
to the previous sample and if required, it can decide to change some of the
interrupt moderation configuration fields. The data sample is composed of data
bandwidth, the number of packets and the number of events. The time between
samples is also measured. Net DIM compares the current and the previous data and
returns an adjusted interrupt moderation configuration object. In some cases,
the algorithm might decide not to change anything. The configuration fields are
the minimum duration (microseconds) allowed between events and the maximum
number of wanted packets per event. The Net DIM algorithm ascribes importance to
increase bandwidth over reducing interrupt rate.

Net DIM Algorithm
=================

Each iteration of the Net DIM algorithm follows these steps:

#. Calculates new data sample.
#. Compares it to previous sample.
#. Makes a decision - suggests interrupt moderation configuration fields.
#. Applies a schedule work function, which applies suggested configuration.

The first two steps are straightforward, both the new and the previous data are
supplied by the driver registered to Net DIM. The previous data is the new data
supplied to the previous iteration. The comparison step checks the difference
between the new and previous data and decides on the result of the last step.
A step would result as "better" if bandwidth increases and as "worse" if
bandwidth reduces. If there is no change in bandwidth, the packet rate is
compared in a similar fashion - increase == "better" and decrease == "worse".
In case there is no change in the packet rate as well, the interrupt rate is
compared. Here the algorithm tries to optimize for lower interrupt rate so an
increase in the interrupt rate is considered "worse" and a decrease is
considered "better". Step #2 has an optimization for avoiding false results: it
only considers a difference between samples as valid if it is greater than a
certain percentage. Also, since Net DIM does not measure anything by itself, it
assumes the data provided by the driver is valid.

Step #3 decides on the suggested configuration based on the result from step #2
and the internal state of the algorithm. The states reflect the "direction" of
the algorithm: is it going left (reducing moderation), right (increasing
moderation) or standing still. Another optimization is that if a decision
to stay still is made multiple times, the interval between iterations of the
algorithm would increase in order to reduce calculation overhead. Also, after
"parking" on one of the most left or most right decisions, the algorithm may
decide to verify this decision by taking a step in the other direction. This is
done in order to avoid getting stuck in a "deep sleep" scenario. Once a
decision is made, an interrupt moderation configuration is selected from
the predefined profiles.

The last step is to notify the registered driver that it should apply the
suggested configuration. This is done by scheduling a work function, defined by
the Net DIM API and provided by the registered driver.

As you can see, Net DIM itself does not actively interact with the system. It
would have trouble making the correct decisions if the wrong data is supplied to
it and it would be useless if the work function would not apply the suggested
configuration. This does, however, allow the registered driver some room for
manoeuvre as it may provide partial data or ignore the algorithm suggestion
under some conditions.

Registering a Network Device to DIM
===================================

Net DIM API exposes the main function net_dim().
This function is the entry point to the Net
DIM algorithm and has to be called every time the driver would like to check if
it should change interrupt moderation parameters. The driver should provide two
data structures: :c:type:`struct dim <dim>` and
:c:type:`struct dim_sample <dim_sample>`. :c:type:`struct dim <dim>`
describes the state of DIM for a specific object (RX queue, TX queue,
other queues, etc.). This includes the current selected profile, previous data
samples, the callback function provided by the driver and more.
:c:type:`struct dim_sample <dim_sample>` describes a data sample,
which will be compared to the data sample stored in :c:type:`struct dim <dim>`
in order to decide on the algorithm's next
step. The sample should include bytes, packets and interrupts, measured by
the driver.

In order to use Net DIM from a networking driver, the driver needs to call the
main net_dim() function. The recommended method is to call net_dim() on each
interrupt. Since Net DIM has a built-in moderation and it might decide to skip
iterations under certain conditions, there is no need to moderate the net_dim()
calls as well. As mentioned above, the driver needs to provide an object of type
:c:type:`struct dim <dim>` to the net_dim() function call. It is advised for
each entity using Net DIM to hold a :c:type:`struct dim <dim>` as part of its
data structure and use it as the main Net DIM API object.
The :c:type:`struct dim_sample <dim_sample>` should hold the latest
bytes, packets and interrupts count. No need to perform any calculations, just
include the raw data.

The net_dim() call itself does not return anything. Instead Net DIM relies on
the driver to provide a callback function, which is called when the algorithm
decides to make a change in the interrupt moderation parameters. This callback
will be scheduled and run in a separate thread in order not to add overhead to
the data flow. After the work is done, Net DIM algorithm needs to be set to
the proper state in order to move to the next iteration.

Example
=======

The following code demonstrates how to register a driver to Net DIM. The actual
usage is not complete but it should make the outline of the usage clear.

.. code-block:: c

  #include <linux/dim.h>

  /* Callback for net DIM to schedule on a decision to change moderation */
  void my_driver_do_dim_work(struct work_struct *work)
  {
	/* Get struct dim from struct work_struct */
	struct dim *dim = container_of(work, struct dim,
				       work);
	/* Do interrupt moderation related stuff */
	...

	/* Signal net DIM work is done and it should move to next iteration */
	dim->state = DIM_START_MEASURE;
  }

  /* My driver's interrupt handler */
  int my_driver_handle_interrupt(struct my_driver_entity *my_entity, ...)
  {
	...
	/* A struct to hold current measured data */
	struct dim_sample dim_sample;
	...
	/* Initiate data sample struct with current data */
	dim_update_sample(my_entity->events,
		          my_entity->packets,
		          my_entity->bytes,
		          &dim_sample);
	/* Call net DIM */
	net_dim(&my_entity->dim, dim_sample);
	...
  }

  /* My entity's initialization function (my_entity was already allocated) */
  int my_driver_init_my_entity(struct my_driver_entity *my_entity, ...)
  {
	...
	/* Initiate struct work_struct with my driver's callback function */
	INIT_WORK(&my_entity->dim.work, my_driver_do_dim_work);
	...
  }

Dynamic Interrupt Moderation (DIM) library API
==============================================

.. kernel-doc:: include/linux/dim.h
    :internal:


code:
.. SPDX-License-Identifier: GPL-2.0

============
NET_FAILOVER
============

Overview
========

The net_failover driver provides an automated failover mechanism via APIs
to create and destroy a failover master netdev and manages a primary and
standby slave netdevs that get registered via the generic failover
infrastructure.

The failover netdev acts a master device and controls 2 slave devices. The
original paravirtual interface is registered as 'standby' slave netdev and
a passthru/vf device with the same MAC gets registered as 'primary' slave
netdev. Both 'standby' and 'failover' netdevs are associated with the same
'pci' device. The user accesses the network interface via 'failover' netdev.
The 'failover' netdev chooses 'primary' netdev as default for transmits when
it is available with link up and running.

This can be used by paravirtual drivers to enable an alternate low latency
datapath. It also enables hypervisor controlled live migration of a VM with
direct attached VF by failing over to the paravirtual datapath when the VF
is unplugged.

virtio-net accelerated datapath: STANDBY mode
=============================================

net_failover enables hypervisor controlled accelerated datapath to virtio-net
enabled VMs in a transparent manner with no/minimal guest userspace changes.

To support this, the hypervisor needs to enable VIRTIO_NET_F_STANDBY
feature on the virtio-net interface and assign the same MAC address to both
virtio-net and VF interfaces.

Here is an example libvirt XML snippet that shows such configuration:
::

  <interface type='network'>
    <mac address='52:54:00:00:12:53'/>
    <source network='enp66s0f0_br'/>
    <target dev='tap01'/>
    <model type='virtio'/>
    <driver name='vhost' queues='4'/>
    <link state='down'/>
    <teaming type='persistent'/>
    <alias name='ua-backup0'/>
  </interface>
  <interface type='hostdev' managed='yes'>
    <mac address='52:54:00:00:12:53'/>
    <source>
      <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
    </source>
    <teaming type='transient' persistent='ua-backup0'/>
  </interface>

In this configuration, the first device definition is for the virtio-net
interface and this acts as the 'persistent' device indicating that this
interface will always be plugged in. This is specified by the 'teaming' tag with
required attribute type having value 'persistent'. The link state for the
virtio-net device is set to 'down' to ensure that the 'failover' netdev prefers
the VF passthrough device for normal communication. The virtio-net device will
be brought UP during live migration to allow uninterrupted communication.

The second device definition is for the VF passthrough interface. Here the
'teaming' tag is provided with type 'transient' indicating that this device may
periodically be unplugged. A second attribute - 'persistent' is provided and
points to the alias name declared for the virtio-net device.

Booting a VM with the above configuration will result in the following 3
interfaces created in the VM:
::

  4: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
      link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff
      inet 192.168.12.53/24 brd 192.168.12.255 scope global dynamic ens10
         valid_lft 42482sec preferred_lft 42482sec
      inet6 fe80::97d8:db2:8c10:b6d6/64 scope link
         valid_lft forever preferred_lft forever
  5: ens10nsby: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master ens10 state DOWN group default qlen 1000
      link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff
  7: ens11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ens10 state UP group default qlen 1000
      link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff

Here, ens10 is the 'failover' master interface, ens10nsby is the slave 'standby'
virtio-net interface, and ens11 is the slave 'primary' VF passthrough interface.

One point to note here is that some user space network configuration daemons
like systemd-networkd, ifupdown, etc, do not understand the 'net_failover'
device; and on the first boot, the VM might end up with both 'failover' device
and VF accquiring IP addresses (either same or different) from the DHCP server.
This will result in lack of connectivity to the VM. So some tweaks might be
needed to these network configuration daemons to make sure that an IP is
received only on the 'failover' device.

Below is the patch snippet used with 'cloud-ifupdown-helper' script found on
Debian cloud images:

::
  @@ -27,6 +27,8 @@ do_setup() {
       local working="$cfgdir/.$INTERFACE"
       local final="$cfgdir/$INTERFACE"

  +    if [ -d "/sys/class/net/${INTERFACE}/master" ]; then exit 0; fi
  +
       if ifup --no-act "$INTERFACE" > /dev/null 2>&1; then
           # interface is already known to ifupdown, no need to generate cfg
           log "Skipping configuration generation for $INTERFACE"

Live Migration of a VM with SR-IOV VF & virtio-net in STANDBY mode
==================================================================

net_failover also enables hypervisor controlled live migration to be supported
with VMs that have direct attached SR-IOV VF devices by automatic failover to
the paravirtual datapath when the VF is unplugged.

Here is a sample script that shows the steps to initiate live migration from
the source hypervisor. Note: It is assumed that the VM is connected to a
software bridge 'br0' which has a single VF attached to it along with the vnet
device to the VM. This is not the VF that was passthrough'd to the VM (seen in
the vf.xml file).
::

  # cat vf.xml
  <interface type='hostdev' managed='yes'>
    <mac address='52:54:00:00:12:53'/>
    <source>
      <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
    </source>
    <teaming type='transient' persistent='ua-backup0'/>
  </interface>

  # Source Hypervisor migrate.sh
  #!/bin/bash

  DOMAIN=vm-01
  PF=ens6np0
  VF=ens6v1             # VF attached to the bridge.
  VF_NUM=1
  TAP_IF=vmtap01        # virtio-net interface in the VM.
  VF_XML=vf.xml

  MAC=52:54:00:00:12:53
  ZERO_MAC=00:00:00:00:00:00

  # Set the virtio-net interface up.
  virsh domif-setlink $DOMAIN $TAP_IF up

  # Remove the VF that was passthrough'd to the VM.
  virsh detach-device --live --config $DOMAIN $VF_XML

  ip link set $PF vf $VF_NUM mac $ZERO_MAC

  # Add FDB entry for traffic to continue going to the VM via
  # the VF -> br0 -> vnet interface path.
  bridge fdb add $MAC dev $VF
  bridge fdb add $MAC dev $TAP_IF master

  # Migrate the VM
  virsh migrate --live --persistent $DOMAIN qemu+ssh://$REMOTE_HOST/system

  # Clean up FDB entries after migration completes.
  bridge fdb del $MAC dev $VF
  bridge fdb del $MAC dev $TAP_IF master

On the destination hypervisor, a shared bridge 'br0' is created before migration
starts, and a VF from the destination PF is added to the bridge. Similarly an
appropriate FDB entry is added.

The following script is executed on the destination hypervisor once migration
completes, and it reattaches the VF to the VM and brings down the virtio-net
interface.

::
  # reattach-vf.sh
  #!/bin/bash

  bridge fdb del 52:54:00:00:12:53 dev ens36v0
  bridge fdb del 52:54:00:00:12:53 dev vmtap01 master
  virsh attach-device --config --live vm01 vf.xml
  virsh domif-setlink vm01 vmtap01 down


code:
.. SPDX-License-Identifier: GPL-2.0

==========
Netconsole
==========

started by Ingo Molnar <mingo@redhat.com>, 2001.09.17

2.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003

IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013

Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015

Please send bug reports to Matt Mackall <mpm@selenic.com>
Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com>

Introduction:
=============

This module logs kernel printk messages over UDP allowing debugging of
problem where disk logging fails and serial consoles are impractical.

It can be used either built-in or as a module. As a built-in,
netconsole initializes immediately after NIC cards and will bring up
the specified interface as soon as possible. While this doesn't allow
capture of early kernel panics, it does capture most of the boot
process.

Sender and receiver configuration:
==================================

It takes a string configuration parameter "netconsole" in the
following format::

 netconsole=[+][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]

   where
	+             if present, enable extended console support
	src-port      source for UDP packets (defaults to 6665)
	src-ip        source IP to use (interface address)
	dev           network interface (eth0)
	tgt-port      port for logging agent (6666)
	tgt-ip        IP address for logging agent
	tgt-macaddr   ethernet MAC address for logging agent (broadcast)

Examples::

 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc

or::

 insmod netconsole netconsole=@/,@10.0.0.2/

or using IPv6::

 insmod netconsole netconsole=@/,@fd00:1:2:3::1/

It also supports logging to multiple remote agents by specifying
parameters for the multiple agents separated by semicolons and the
complete string enclosed in "quotes", thusly::

 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/"

Built-in netconsole starts immediately after the TCP stack is
initialized and attempts to bring up the supplied dev at the supplied
address.

The remote host has several options to receive the kernel messages,
for example:

1) syslogd

2) netcat

   On distributions using a BSD-based netcat version (e.g. Fedora,
   openSUSE and Ubuntu) the listening port must be specified without
   the -p switch::

	nc -u -l -p <port>' / 'nc -u -l <port>

    or::

	netcat -u -l -p <port>' / 'netcat -u -l <port>

3) socat

::

   socat udp-recv:<port> -

Dynamic reconfiguration:
========================

Dynamic reconfigurability is a useful addition to netconsole that enables
remote logging targets to be dynamically added, removed, or have their
parameters reconfigured at runtime from a configfs-based userspace interface.
[ Note that the parameters of netconsole targets that were specified/created
from the boot/module option are not exposed via this interface, and hence
cannot be modified dynamically. ]

To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the
netconsole module (or kernel, if netconsole is built-in).

Some examples follow (where configfs is mounted at the /sys/kernel/config
mountpoint).

To add a remote logging target (target names can be arbitrary)::

 cd /sys/kernel/config/netconsole/
 mkdir target1

Note that newly created targets have default parameter values (as mentioned
above) and are disabled by default -- they must first be enabled by writing
"1" to the "enabled" attribute (usually after setting parameters accordingly)
as described below.

To remove a target::

 rmdir /sys/kernel/config/netconsole/othertarget/

The interface exposes these parameters of a netconsole target to userspace:

	==============  =================================       ============
	enabled		Is this target currently enabled?	(read-write)
	extended	Extended mode enabled			(read-write)
	dev_name	Local network interface name		(read-write)
	local_port	Source UDP port to use			(read-write)
	remote_port	Remote agent's UDP port			(read-write)
	local_ip	Source IP address to use		(read-write)
	remote_ip	Remote agent's IP address		(read-write)
	local_mac	Local interface's MAC address		(read-only)
	remote_mac	Remote agent's MAC address		(read-write)
	==============  =================================       ============

The "enabled" attribute is also used to control whether the parameters of
a target can be updated or not -- you can modify the parameters of only
disabled targets (i.e. if "enabled" is 0).

To update a target's parameters::

 cat enabled				# check if enabled is 1
 echo 0 > enabled			# disable the target (if required)
 echo eth2 > dev_name			# set local interface
 echo 10.0.0.4 > remote_ip		# update some parameter
 echo cb:a9:87:65:43:21 > remote_mac	# update more parameters
 echo 1 > enabled			# enable target again

You can also update the local interface dynamically. This is especially
useful if you want to use interfaces that have newly come up (and may not
have existed when netconsole was loaded / initialized).

Extended console:
=================

If '+' is prefixed to the configuration line or "extended" config file
is set to 1, extended console support is enabled. An example boot
param follows::

 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc

Log messages are transmitted with extended metadata header in the
following format which is the same as /dev/kmsg::

 <level>,<sequnum>,<timestamp>,<contflag>;<message text>

Non printable characters in <message text> are escaped using "\xff"
notation. If the message contains optional dictionary, verbatim
newline is used as the delimeter.

If a message doesn't fit in certain number of bytes (currently 1000),
the message is split into multiple fragments by netconsole. These
fragments are transmitted with "ncfrag" header field added::

 ncfrag=<byte-offset>/<total-bytes>

For example, assuming a lot smaller chunk size, a message "the first
chunk, the 2nd chunk." may be split as follows::

 6,416,1758426,-,ncfrag=0/31;the first chunk,
 6,416,1758426,-,ncfrag=16/31; the 2nd chunk.

Miscellaneous notes:
====================

.. Warning::

   the default target ethernet setting uses the broadcast
   ethernet address to send packets, which can cause increased load on
   other systems on the same ethernet segment.

.. Tip::

   some LAN switches may be configured to suppress ethernet broadcasts
   so it is advised to explicitly specify the remote agents' MAC addresses
   from the config parameters passed to netconsole.

.. Tip::

   to find out the MAC address of, say, 10.0.0.2, you may try using::

	ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2

.. Tip::

   in case the remote logging agent is on a separate LAN subnet than
   the sender, it is suggested to try specifying the MAC address of the
   default gateway (you may use /sbin/route -n to find it out) as the
   remote MAC address instead.

.. note::

   the network device (eth1 in the above case) can run any kind
   of other network traffic, netconsole is not intrusive. Netconsole
   might cause slight delays in other traffic if the volume of kernel
   messages is high, but should have no other impact.

.. note::

   if you find that the remote logging agent is not receiving or
   printing all messages from the sender, it is likely that you have set
   the "console_loglevel" parameter (on the sender) to only send high
   priority messages to the console. You can change this at runtime using::

	dmesg -n 8

   or by specifying "debug" on the kernel command line at boot, to send
   all kernel messages to the console. A specific value for this parameter
   can also be set using the "loglevel" kernel boot option. See the
   dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst
   for details.

Netconsole was designed to be as instantaneous as possible, to
enable the logging of even the most critical kernel bugs. It works
from IRQ contexts as well, and does not enable interrupts while
sending packets. Due to these unique needs, configuration cannot
be more automatic, and some fundamental limitations will remain:
only IP networks, UDP packets and ethernet devices are supported.


code:
.. SPDX-License-Identifier: GPL-2.0

=====================================================
Netdev features mess and how to get out from it alive
=====================================================

Author:
	Michał Mirosław <mirq-linux@rere.qmqm.pl>

Part I: Feature sets
====================

Long gone are the days when a network card would just take and give packets
verbatim.  Today's devices add multiple features and bugs (read: offloads)
that relieve an OS of various tasks like generating and checking checksums,
splitting packets, classifying them.  Those capabilities and their state
are commonly referred to as netdev features in Linux kernel world.

There are currently three sets of features relevant to the driver, and
one used internally by network core:

 1. netdev->hw_features set contains features whose state may possibly
    be changed (enabled or disabled) for a particular device by user's
    request.  This set should be initialized in ndo_init callback and not
    changed later.

 2. netdev->features set contains features which are currently enabled
    for a device.  This should be changed only by network core or in
    error paths of ndo_set_features callback.

 3. netdev->vlan_features set contains features whose state is inherited
    by child VLAN devices (limits netdev->features set).  This is currently
    used for all VLAN devices whether tags are stripped or inserted in
    hardware or software.

 4. netdev->wanted_features set contains feature set requested by user.
    This set is filtered by ndo_fix_features callback whenever it or
    some device-specific conditions change. This set is internal to
    networking core and should not be referenced in drivers.

Part II: Controlling enabled features
=====================================

When current feature set (netdev->features) is to be changed, new set
is calculated and filtered by calling ndo_fix_features callback
and netdev_fix_features(). If the resulting set differs from current
set, it is passed to ndo_set_features callback and (if the callback
returns success) replaces value stored in netdev->features.
NETDEV_FEAT_CHANGE notification is issued after that whenever current
set might have changed.

The following events trigger recalculation:
 1. device's registration, after ndo_init returned success
 2. user requested changes in features state
 3. netdev_update_features() is called

ndo_*_features callbacks are called with rtnl_lock held. Missing callbacks
are treated as always returning success.

A driver that wants to trigger recalculation must do so by calling
netdev_update_features() while holding rtnl_lock. This should not be done
from ndo_*_features callbacks. netdev->features should not be modified by
driver except by means of ndo_fix_features callback.

Part III: Implementation hints
==============================

 * ndo_fix_features:

All dependencies between features should be resolved here. The resulting
set can be reduced further by networking core imposed limitations (as coded
in netdev_fix_features()). For this reason it is safer to disable a feature
when its dependencies are not met instead of forcing the dependency on.

This callback should not modify hardware nor driver state (should be
stateless).  It can be called multiple times between successive
ndo_set_features calls.

Callback must not alter features contained in NETIF_F_SOFT_FEATURES or
NETIF_F_NEVER_CHANGE sets. The exception is NETIF_F_VLAN_CHALLENGED but
care must be taken as the change won't affect already configured VLANs.

 * ndo_set_features:

Hardware should be reconfigured to match passed feature set. The set
should not be altered unless some error condition happens that can't
be reliably detected in ndo_fix_features. In this case, the callback
should update netdev->features to match resulting hardware state.
Errors returned are not (and cannot be) propagated anywhere except dmesg.
(Note: successful return is zero, >0 means silent error.)

Part IV: Features
=================

For current list of features, see include/linux/netdev_features.h.
This section describes semantics of some of them.

 * Transmit checksumming

For complete description, see comments near the top of include/linux/skbuff.h.

Note: NETIF_F_HW_CSUM is a superset of NETIF_F_IP_CSUM + NETIF_F_IPV6_CSUM.
It means that device can fill TCP/UDP-like checksum anywhere in the packets
whatever headers there might be.

 * Transmit TCP segmentation offload

NETIF_F_TSO_ECN means that hardware can properly split packets with CWR bit
set, be it TCPv4 (when NETIF_F_TSO is enabled) or TCPv6 (NETIF_F_TSO6).

 * Transmit UDP segmentation offload

NETIF_F_GSO_UDP_L4 accepts a single UDP header with a payload that exceeds
gso_size. On segmentation, it segments the payload on gso_size boundaries and
replicates the network and UDP headers (fixing up the last one if less than
gso_size).

 * Transmit DMA from high memory

On platforms where this is relevant, NETIF_F_HIGHDMA signals that
ndo_start_xmit can handle skbs with frags in high memory.

 * Transmit scatter-gather

Those features say that ndo_start_xmit can handle fragmented skbs:
NETIF_F_SG --- paged skbs (skb_shinfo()->frags), NETIF_F_FRAGLIST ---
chained skbs (skb->next/prev list).

 * Software features

Features contained in NETIF_F_SOFT_FEATURES are features of networking
stack. Driver should not change behaviour based on them.

 * LLTX driver (deprecated for hardware drivers)

NETIF_F_LLTX is meant to be used by drivers that don't need locking at all,
e.g. software tunnels.

This is also used in a few legacy drivers that implement their
own locking, don't use it for new (hardware) drivers.

 * netns-local device

NETIF_F_NETNS_LOCAL is set for devices that are not allowed to move between
network namespaces (e.g. loopback).

Don't use it in drivers.

 * VLAN challenged

NETIF_F_VLAN_CHALLENGED should be set for devices which can't cope with VLAN
headers. Some drivers set this because the cards can't handle the bigger MTU.
[FIXME: Those cases could be fixed in VLAN code by allowing only reduced-MTU
VLANs. This may be not useful, though.]

*  rx-fcs

This requests that the NIC append the Ethernet Frame Checksum (FCS)
to the end of the skb data.  This allows sniffers and other tools to
read the CRC recorded by the NIC on receipt of the packet.

*  rx-all

This requests that the NIC receive all possible frames, including errored
frames (such as bad FCS, etc).  This can be helpful when sniffing a link with
bad packets on it.  Some NICs may receive more packets if also put into normal
PROMISC mode.

*  rx-gro-hw

This requests that the NIC enables Hardware GRO (generic receive offload).
Hardware GRO is basically the exact reverse of TSO, and is generally
stricter than Hardware LRO.  A packet stream merged by Hardware GRO must
be re-segmentable by GSO or TSO back to the exact original packet stream.
Hardware GRO is dependent on RXCSUM since every packet successfully merged
by hardware must also have the checksum verified by hardware.

* hsr-tag-ins-offload

This should be set for devices which insert an HSR (High-availability Seamless
Redundancy) or PRP (Parallel Redundancy Protocol) tag automatically.

* hsr-tag-rm-offload

This should be set for devices which remove HSR (High-availability Seamless
Redundancy) or PRP (Parallel Redundancy Protocol) tags automatically.

* hsr-fwd-offload

This should be set for devices which forward HSR (High-availability Seamless
Redundancy) frames from one port to another in hardware.

* hsr-dup-offload

This should be set for devices which duplicate outgoing HSR (High-availability
Seamless Redundancy) or PRP (Parallel Redundancy Protocol) tags automatically
frames in hardware.


code:
.. SPDX-License-Identifier: GPL-2.0

=====================================
Network Devices, the Kernel, and You!
=====================================

Introduction
============
The following is a random collection of documentation regarding
network devices.

struct net_device lifetime rules
================================
Network device structures need to persist even after module is unloaded and
must be allocated with alloc_netdev_mqs() and friends.
If device has registered successfully, it will be freed on last use
by free_netdev(). This is required to handle the pathological case cleanly
(example: ``rmmod mydriver </sys/class/net/myeth/mtu``)

alloc_netdev_mqs() / alloc_netdev() reserve extra space for driver
private data which gets freed when the network device is freed. If
separately allocated data is attached to the network device
(netdev_priv()) then it is up to the module exit handler to free that.

There are two groups of APIs for registering struct net_device.
First group can be used in normal contexts where ``rtnl_lock`` is not already
held: register_netdev(), unregister_netdev().
Second group can be used when ``rtnl_lock`` is already held:
register_netdevice(), unregister_netdevice(), free_netdevice().

Simple drivers
--------------

Most drivers (especially device drivers) handle lifetime of struct net_device
in context where ``rtnl_lock`` is not held (e.g. driver probe and remove paths).

In that case the struct net_device registration is done using
the register_netdev(), and unregister_netdev() functions:

.. code-block:: c

  int probe()
  {
    struct my_device_priv *priv;
    int err;

    dev = alloc_netdev_mqs(...);
    if (!dev)
      return -ENOMEM;
    priv = netdev_priv(dev);

    /* ... do all device setup before calling register_netdev() ...
     */

    err = register_netdev(dev);
    if (err)
      goto err_undo;

    /* net_device is visible to the user! */

  err_undo:
    /* ... undo the device setup ... */
    free_netdev(dev);
    return err;
  }

  void remove()
  {
    unregister_netdev(dev);
    free_netdev(dev);
  }

Note that after calling register_netdev() the device is visible in the system.
Users can open it and start sending / receiving traffic immediately,
or run any other callback, so all initialization must be done prior to
registration.

unregister_netdev() closes the device and waits for all users to be done
with it. The memory of struct net_device itself may still be referenced
by sysfs but all operations on that device will fail.

free_netdev() can be called after unregister_netdev() returns on when
register_netdev() failed.

Device management under RTNL
----------------------------

Registering struct net_device while in context which already holds
the ``rtnl_lock`` requires extra care. In those scenarios most drivers
will want to make use of struct net_device's ``needs_free_netdev``
and ``priv_destructor`` members for freeing of state.

Example flow of netdev handling under ``rtnl_lock``:

.. code-block:: c

  static void my_setup(struct net_device *dev)
  {
    dev->needs_free_netdev = true;
  }

  static void my_destructor(struct net_device *dev)
  {
    some_obj_destroy(priv->obj);
    some_uninit(priv);
  }

  int create_link()
  {
    struct my_device_priv *priv;
    int err;

    ASSERT_RTNL();

    dev = alloc_netdev(sizeof(*priv), "net%d", NET_NAME_UNKNOWN, my_setup);
    if (!dev)
      return -ENOMEM;
    priv = netdev_priv(dev);

    /* Implicit constructor */
    err = some_init(priv);
    if (err)
      goto err_free_dev;

    priv->obj = some_obj_create();
    if (!priv->obj) {
      err = -ENOMEM;
      goto err_some_uninit;
    }
    /* End of constructor, set the destructor: */
    dev->priv_destructor = my_destructor;

    err = register_netdevice(dev);
    if (err)
      /* register_netdevice() calls destructor on failure */
      goto err_free_dev;

    /* If anything fails now unregister_netdevice() (or unregister_netdev())
     * will take care of calling my_destructor and free_netdev().
     */

    return 0;

  err_some_uninit:
    some_uninit(priv);
  err_free_dev:
    free_netdev(dev);
    return err;
  }

If struct net_device.priv_destructor is set it will be called by the core
some time after unregister_netdevice(), it will also be called if
register_netdevice() fails. The callback may be invoked with or without
``rtnl_lock`` held.

There is no explicit constructor callback, driver "constructs" the private
netdev state after allocating it and before registration.

Setting struct net_device.needs_free_netdev makes core call free_netdevice()
automatically after unregister_netdevice() when all references to the device
are gone. It only takes effect after a successful call to register_netdevice()
so if register_netdevice() fails driver is responsible for calling
free_netdev().

free_netdev() is safe to call on error paths right after unregister_netdevice()
or when register_netdevice() fails. Parts of netdev (de)registration process
happen after ``rtnl_lock`` is released, therefore in those cases free_netdev()
will defer some of the processing until ``rtnl_lock`` is released.

Devices spawned from struct rtnl_link_ops should never free the
struct net_device directly.

.ndo_init and .ndo_uninit
~~~~~~~~~~~~~~~~~~~~~~~~~

``.ndo_init`` and ``.ndo_uninit`` callbacks are called during net_device
registration and de-registration, under ``rtnl_lock``. Drivers can use
those e.g. when parts of their init process need to run under ``rtnl_lock``.

``.ndo_init`` runs before device is visible in the system, ``.ndo_uninit``
runs during de-registering after device is closed but other subsystems
may still have outstanding references to the netdevice.

MTU
===
Each network device has a Maximum Transfer Unit. The MTU does not
include any link layer protocol overhead. Upper layer protocols must
not pass a socket buffer (skb) to a device to transmit with more data
than the mtu. The MTU does not include link layer header overhead, so
for example on Ethernet if the standard MTU is 1500 bytes used, the
actual skb will contain up to 1514 bytes because of the Ethernet
header. Devices should allow for the 4 byte VLAN header as well.

Segmentation Offload (GSO, TSO) is an exception to this rule.  The
upper layer protocol may pass a large socket buffer to the device
transmit routine, and the device will break that up into separate
packets based on the current MTU.

MTU is symmetrical and applies both to receive and transmit. A device
must be able to receive at least the maximum size packet allowed by
the MTU. A network device may use the MTU as mechanism to size receive
buffers, but the device should allow packets with VLAN header. With
standard Ethernet mtu of 1500 bytes, the device should allow up to
1518 byte packets (1500 + 14 header + 4 tag).  The device may either:
drop, truncate, or pass up oversize packets, but dropping oversize
packets is preferred.

struct net_device synchronization rules
=======================================
ndo_open:
	Synchronization: rtnl_lock() semaphore.
	Context: process

ndo_stop:
	Synchronization: rtnl_lock() semaphore.
	Context: process
	Note: netif_running() is guaranteed false

ndo_do_ioctl:
	Synchronization: rtnl_lock() semaphore.
	Context: process

        This is only called by network subsystems internally,
        not by user space calling ioctl as it was in before
        linux-5.14.

ndo_siocbond:
        Synchronization: rtnl_lock() semaphore.
        Context: process

        Used by the bonding driver for the SIOCBOND family of
        ioctl commands.

ndo_siocwandev:
	Synchronization: rtnl_lock() semaphore.
	Context: process

	Used by the drivers/net/wan framework to handle
	the SIOCWANDEV ioctl with the if_settings structure.

ndo_siocdevprivate:
	Synchronization: rtnl_lock() semaphore.
	Context: process

	This is used to implement SIOCDEVPRIVATE ioctl helpers.
	These should not be added to new drivers, so don't use.

ndo_eth_ioctl:
	Synchronization: rtnl_lock() semaphore.
	Context: process

ndo_get_stats:
	Synchronization: rtnl_lock() semaphore, dev_base_lock rwlock, or RCU.
	Context: atomic (can't sleep under rwlock or RCU)

ndo_start_xmit:
	Synchronization: __netif_tx_lock spinlock.

	When the driver sets NETIF_F_LLTX in dev->features this will be
	called without holding netif_tx_lock. In this case the driver
	has to lock by itself when needed.
	The locking there should also properly protect against
	set_rx_mode. WARNING: use of NETIF_F_LLTX is deprecated.
	Don't use it for new drivers.

	Context: Process with BHs disabled or BH (timer),
		 will be called with interrupts disabled by netconsole.

	Return codes:

	* NETDEV_TX_OK everything ok.
	* NETDEV_TX_BUSY Cannot transmit packet, try later
	  Usually a bug, means queue start/stop flow control is broken in
	  the driver. Note: the driver must NOT put the skb in its DMA ring.

ndo_tx_timeout:
	Synchronization: netif_tx_lock spinlock; all TX queues frozen.
	Context: BHs disabled
	Notes: netif_queue_stopped() is guaranteed true

ndo_set_rx_mode:
	Synchronization: netif_addr_lock spinlock.
	Context: BHs disabled

struct napi_struct synchronization rules
========================================
napi->poll:
	Synchronization:
		NAPI_STATE_SCHED bit in napi->state.  Device
		driver's ndo_stop method will invoke napi_disable() on
		all NAPI instances which will do a sleeping poll on the
		NAPI_STATE_SCHED napi->state bit, waiting for all pending
		NAPI activity to cease.

	Context:
		 softirq
		 will be called with interrupts disabled by netconsole.


code:
.. SPDX-License-Identifier: GPL-2.0

=========================
Netfilter Sysfs variables
=========================

/proc/sys/net/netfilter/* Variables:
====================================

nf_log_all_netns - BOOLEAN
	- 0 - disabled (default)
	- not 0 - enabled

	By default, only init_net namespace can log packets into kernel log
	with LOG target; this aims to prevent containers from flooding host
	kernel log. If enabled, this target also works in other network
	namespaces. This variable is only accessible from init_net.


code:
.. SPDX-License-Identifier: GPL-2.0

===============
NETIF Msg Level
===============

The design of the network interface message level setting.

History
-------

 The design of the debugging message interface was guided and
 constrained by backwards compatibility previous practice.  It is useful
 to understand the history and evolution in order to understand current
 practice and relate it to older driver source code.

 From the beginning of Linux, each network device driver has had a local
 integer variable that controls the debug message level.  The message
 level ranged from 0 to 7, and monotonically increased in verbosity.

 The message level was not precisely defined past level 3, but were
 always implemented within +-1 of the specified level.  Drivers tended
 to shed the more verbose level messages as they matured.

   - 0  Minimal messages, only essential information on fatal errors.
   - 1  Standard messages, initialization status.  No run-time messages
   - 2  Special media selection messages, generally timer-driver.
   - 3  Interface starts and stops, including normal status messages
   - 4  Tx and Rx frame error messages, and abnormal driver operation
   - 5  Tx packet queue information, interrupt events.
   - 6  Status on each completed Tx packet and received Rx packets
   - 7  Initial contents of Tx and Rx packets

 Initially this message level variable was uniquely named in each driver
 e.g. "lance_debug", so that a kernel symbolic debugger could locate and
 modify the setting.  When kernel modules became common, the variables
 were consistently renamed to "debug" and allowed to be set as a module
 parameter.

 This approach worked well.  However there is always a demand for
 additional features.  Over the years the following emerged as
 reasonable and easily implemented enhancements

   - Using an ioctl() call to modify the level.
   - Per-interface rather than per-driver message level setting.
   - More selective control over the type of messages emitted.

 The netif_msg recommendation adds these features with only a minor
 complexity and code size increase.

 The recommendation is the following points

  - Retaining the per-driver integer variable "debug" as a module
    parameter with a default level of '1'.

  - Adding a per-interface private variable named "msg_enable".  The
    variable is a bit map rather than a level, and is initialized as::

       1 << debug

    Or more precisely::

	debug < 0 ? 0 : 1 << min(sizeof(int)-1, debug)

    Messages should changes from::

      if (debug > 1)
	   printk(MSG_DEBUG "%s: ...

    to::

      if (np->msg_enable & NETIF_MSG_LINK)
	   printk(MSG_DEBUG "%s: ...

The set of message levels is named

  =========   ===================	============
  Old level   Name			Bit position
  =========   ===================	============
    0         NETIF_MSG_DRV		0x0001
    1         NETIF_MSG_PROBE		0x0002
    2         NETIF_MSG_LINK		0x0004
    2         NETIF_MSG_TIMER		0x0004
    3         NETIF_MSG_IFDOWN		0x0008
    3         NETIF_MSG_IFUP		0x0008
    4         NETIF_MSG_RX_ERR		0x0010
    4         NETIF_MSG_TX_ERR		0x0010
    5         NETIF_MSG_TX_QUEUED	0x0020
    5         NETIF_MSG_INTR		0x0020
    6         NETIF_MSG_TX_DONE		0x0040
    6         NETIF_MSG_RX_STATUS	0x0040
    7         NETIF_MSG_PKTDATA		0x0080
  =========   ===================	============


code:
.. SPDX-License-Identifier: GPL-2.0

=========================
Resilient Next-hop Groups
=========================

Resilient groups are a type of next-hop group that is aimed at minimizing
disruption in flow routing across changes to the group composition and
weights of constituent next hops.

The idea behind resilient hashing groups is best explained in contrast to
the legacy multipath next-hop group, which uses the hash-threshold
algorithm, described in RFC 2992.

To select a next hop, hash-threshold algorithm first assigns a range of
hashes to each next hop in the group, and then selects the next hop by
comparing the SKB hash with the individual ranges. When a next hop is
removed from the group, the ranges are recomputed, which leads to
reassignment of parts of hash space from one next hop to another. RFC 2992
illustrates it thus::

             +-------+-------+-------+-------+-------+
             |   1   |   2   |   3   |   4   |   5   |
             +-------+-+-----+---+---+-----+-+-------+
             |    1    |    2    |    4    |    5    |
             +---------+---------+---------+---------+

              Before and after deletion of next hop 3
	      under the hash-threshold algorithm.

Note how next hop 2 gave up part of the hash space in favor of next hop 1,
and 4 in favor of 5. While there will usually be some overlap between the
previous and the new distribution, some traffic flows change the next hop
that they resolve to.

If a multipath group is used for load-balancing between multiple servers,
this hash space reassignment causes an issue that packets from a single
flow suddenly end up arriving at a server that does not expect them. This
can result in TCP connections being reset.

If a multipath group is used for load-balancing among available paths to
the same server, the issue is that different latencies and reordering along
the way causes the packets to arrive in the wrong order, resulting in
degraded application performance.

To mitigate the above-mentioned flow redirection, resilient next-hop groups
insert another layer of indirection between the hash space and its
constituent next hops: a hash table. The selection algorithm uses SKB hash
to choose a hash table bucket, then reads the next hop that this bucket
contains, and forwards traffic there.

This indirection brings an important feature. In the hash-threshold
algorithm, the range of hashes associated with a next hop must be
continuous. With a hash table, mapping between the hash table buckets and
the individual next hops is arbitrary. Therefore when a next hop is deleted
the buckets that held it are simply reassigned to other next hops::

	    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |1|1|1|1|2|2|2|2|3|3|3|3|4|4|4|4|5|5|5|5|
	    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	                     v v v v
	    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |1|1|1|1|2|2|2|2|1|2|4|5|4|4|4|4|5|5|5|5|
	    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

	    Before and after deletion of next hop 3
	    under the resilient hashing algorithm.

When weights of next hops in a group are altered, it may be possible to
choose a subset of buckets that are currently not used for forwarding
traffic, and use those to satisfy the new next-hop distribution demands,
keeping the "busy" buckets intact. This way, established flows are ideally
kept being forwarded to the same endpoints through the same paths as before
the next-hop group change.

Algorithm
---------

In a nutshell, the algorithm works as follows. Each next hop deserves a
certain number of buckets, according to its weight and the number of
buckets in the hash table. In accordance with the source code, we will call
this number a "wants count" of a next hop. In case of an event that might
cause bucket allocation change, the wants counts for individual next hops
are updated.

Next hops that have fewer buckets than their wants count, are called
"underweight". Those that have more are "overweight". If there are no
overweight (and therefore no underweight) next hops in the group, it is
said to be "balanced".

Each bucket maintains a last-used timer. Every time a packet is forwarded
through a bucket, this timer is updated to current jiffies value. One
attribute of a resilient group is then the "idle timer", which is the
amount of time that a bucket must not be hit by traffic in order for it to
be considered "idle". Buckets that are not idle are busy.

After assigning wants counts to next hops, an "upkeep" algorithm runs. For
buckets:

1) that have no assigned next hop, or
2) whose next hop has been removed, or
3) that are idle and their next hop is overweight,

upkeep changes the next hop that the bucket references to one of the
underweight next hops. If, after considering all buckets in this manner,
there are still underweight next hops, another upkeep run is scheduled to a
future time.

There may not be enough "idle" buckets to satisfy the updated wants counts
of all next hops. Another attribute of a resilient group is the "unbalanced
timer". This timer can be set to 0, in which case the table will stay out
of balance until idle buckets do appear, possibly never. If set to a
non-zero value, the value represents the period of time that the table is
permitted to stay out of balance.

With this in mind, we update the above list of conditions with one more
item. Thus buckets:

4) whose next hop is overweight, and the amount of time that the table has
   been out of balance exceeds the unbalanced timer, if that is non-zero,

\... are migrated as well.

Offloading & Driver Feedback
----------------------------

When offloading resilient groups, the algorithm that distributes buckets
among next hops is still the one in SW. Drivers are notified of updates to
next hop groups in the following three ways:

- Full group notification with the type
  ``NH_NOTIFIER_INFO_TYPE_RES_TABLE``. This is used just after the group is
  created and buckets populated for the first time.

- Single-bucket notifications of the type
  ``NH_NOTIFIER_INFO_TYPE_RES_BUCKET``, which is used for notifications of
  individual migrations within an already-established group.

- Pre-replace notification, ``NEXTHOP_EVENT_RES_TABLE_PRE_REPLACE``. This
  is sent before the group is replaced, and is a way for the driver to veto
  the group before committing anything to the HW.

Some single-bucket notifications are forced, as indicated by the "force"
flag in the notification. Those are used for the cases where e.g. the next
hop associated with the bucket was removed, and the bucket really must be
migrated.

Non-forced notifications can be overridden by the driver by returning an
error code. The use case for this is that the driver notifies the HW that a
bucket should be migrated, but the HW discovers that the bucket has in fact
been hit by traffic.

A second way for the HW to report that a bucket is busy is through the
``nexthop_res_grp_activity_update()`` API. The buckets identified this way
as busy are treated as if traffic hit them.

Offloaded buckets should be flagged as either "offload" or "trap". This is
done through the ``nexthop_bucket_set_hw_flags()`` API.

Netlink UAPI
------------

Resilient Group Replacement
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Resilient groups are configured using the ``RTM_NEWNEXTHOP`` message in the
same manner as other multipath groups. The following changes apply to the
attributes passed in the netlink message:

  =================== =========================================================
  ``NHA_GROUP_TYPE``  Should be ``NEXTHOP_GRP_TYPE_RES`` for resilient group.
  ``NHA_RES_GROUP``   A nest that contains attributes specific to resilient
                      groups.
  =================== =========================================================

``NHA_RES_GROUP`` payload:

  =================================== =========================================
  ``NHA_RES_GROUP_BUCKETS``           Number of buckets in the hash table.
  ``NHA_RES_GROUP_IDLE_TIMER``        Idle timer in units of clock_t.
  ``NHA_RES_GROUP_UNBALANCED_TIMER``  Unbalanced timer in units of clock_t.
  =================================== =========================================

Next Hop Get
^^^^^^^^^^^^

Requests to get resilient next-hop groups use the ``RTM_GETNEXTHOP``
message in exactly the same way as other next hop get requests. The
response attributes match the replacement attributes cited above, except
``NHA_RES_GROUP`` payload will include the following attribute:

  =================================== =========================================
  ``NHA_RES_GROUP_UNBALANCED_TIME``   How long has the resilient group been out
                                      of balance, in units of clock_t.
  =================================== =========================================

Bucket Get
^^^^^^^^^^

The message ``RTM_GETNEXTHOPBUCKET`` without the ``NLM_F_DUMP`` flag is
used to request a single bucket. The attributes recognized at get requests
are:

  =================== =========================================================
  ``NHA_ID``          ID of the next-hop group that the bucket belongs to.
  ``NHA_RES_BUCKET``  A nest that contains attributes specific to bucket.
  =================== =========================================================

``NHA_RES_BUCKET`` payload:

  ======================== ====================================================
  ``NHA_RES_BUCKET_INDEX`` Index of bucket in the resilient table.
  ======================== ====================================================

Bucket Dumps
^^^^^^^^^^^^

The message ``RTM_GETNEXTHOPBUCKET`` with the ``NLM_F_DUMP`` flag is used
to request a dump of matching buckets. The attributes recognized at dump
requests are:

  =================== =========================================================
  ``NHA_ID``          If specified, limits the dump to just the next-hop group
                      with this ID.
  ``NHA_OIF``         If specified, limits the dump to buckets that contain
                      next hops that use the device with this ifindex.
  ``NHA_MASTER``      If specified, limits the dump to buckets that contain
                      next hops that use a device in the VRF with this ifindex.
  ``NHA_RES_BUCKET``  A nest that contains attributes specific to bucket.
  =================== =========================================================

``NHA_RES_BUCKET`` payload:

  ======================== ====================================================
  ``NHA_RES_BUCKET_NH_ID`` If specified, limits the dump to just the buckets
                           that contain the next hop with this ID.
  ======================== ====================================================

Usage
-----

To illustrate the usage, consider the following commands::

	# ip nexthop add id 1 via 192.0.2.2 dev eth0
	# ip nexthop add id 2 via 192.0.2.3 dev eth0
	# ip nexthop add id 10 group 1/2 type resilient \
		buckets 8 idle_timer 60 unbalanced_timer 300

The last command creates a resilient next-hop group. It will have 8 buckets
(which is unusually low number, and used here for demonstration purposes
only), each bucket will be considered idle when no traffic hits it for at
least 60 seconds, and if the table remains out of balance for 300 seconds,
it will be forcefully brought into balance.

Changing next-hop weights leads to change in bucket allocation::

	# ip nexthop replace id 10 group 1,3/2 type resilient

This can be confirmed by looking at individual buckets::

	# ip nexthop bucket show id 10
	id 10 index 0 idle_time 5.59 nhid 1
	id 10 index 1 idle_time 5.59 nhid 1
	id 10 index 2 idle_time 8.74 nhid 2
	id 10 index 3 idle_time 8.74 nhid 2
	id 10 index 4 idle_time 8.74 nhid 1
	id 10 index 5 idle_time 8.74 nhid 1
	id 10 index 6 idle_time 8.74 nhid 1
	id 10 index 7 idle_time 8.74 nhid 1

Note the two buckets that have a shorter idle time. Those are the ones that
were migrated after the next-hop replace command to satisfy the new demand
that next hop 1 be given 6 buckets instead of 4.

Netdevsim
---------

The netdevsim driver implements a mock offload of resilient groups, and
exposes debugfs interface that allows marking individual buckets as busy.
For example, the following will mark bucket 23 in next-hop group 10 as
active::

	# echo 10 23 > /sys/kernel/debug/netdevsim/netdevsim10/fib/nexthop_bucket_activity

In addition, another debugfs interface can be used to configure that the
next attempt to migrate a bucket should fail::

	# echo 1 > /sys/kernel/debug/netdevsim/netdevsim10/fib/fail_nexthop_bucket_replace

Besides serving as an example, the interfaces that netdevsim exposes are
useful in automated testing, and
``tools/testing/selftests/drivers/net/netdevsim/nexthop.sh`` makes use of
them to test the algorithm.


code:
.. SPDX-License-Identifier: GPL-2.0

===================================
Netfilter Conntrack Sysfs variables
===================================

/proc/sys/net/netfilter/nf_conntrack_* Variables:
=================================================

nf_conntrack_acct - BOOLEAN
	- 0 - disabled (default)
	- not 0 - enabled

	Enable connection tracking flow accounting. 64-bit byte and packet
	counters per flow are added.

nf_conntrack_buckets - INTEGER
	Size of hash table. If not specified as parameter during module
	loading, the default size is calculated by dividing total memory
	by 16384 to determine the number of buckets. The hash table will
	never have fewer than 1024 and never more than 262144 buckets.
	This sysctl is only writeable in the initial net namespace.

nf_conntrack_checksum - BOOLEAN
	- 0 - disabled
	- not 0 - enabled (default)

	Verify checksum of incoming packets. Packets with bad checksums are
	in INVALID state. If this is enabled, such packets will not be
	considered for connection tracking.

nf_conntrack_count - INTEGER (read-only)
	Number of currently allocated flow entries.

nf_conntrack_events - BOOLEAN
	- 0 - disabled
	- 1 - enabled
	- 2 - auto (default)

	If this option is enabled, the connection tracking code will
	provide userspace with connection tracking events via ctnetlink.
	The default allocates the extension if a userspace program is
	listening to ctnetlink events.

nf_conntrack_expect_max - INTEGER
	Maximum size of expectation table.  Default value is
	nf_conntrack_buckets / 256. Minimum is 1.

nf_conntrack_frag6_high_thresh - INTEGER
	default 262144

	Maximum memory used to reassemble IPv6 fragments.  When
	nf_conntrack_frag6_high_thresh bytes of memory is allocated for this
	purpose, the fragment handler will toss packets until
	nf_conntrack_frag6_low_thresh is reached.

nf_conntrack_frag6_low_thresh - INTEGER
	default 196608

	See nf_conntrack_frag6_low_thresh

nf_conntrack_frag6_timeout - INTEGER (seconds)
	default 60

	Time to keep an IPv6 fragment in memory.

nf_conntrack_generic_timeout - INTEGER (seconds)
	default 600

	Default for generic timeout.  This refers to layer 4 unknown/unsupported
	protocols.

nf_conntrack_icmp_timeout - INTEGER (seconds)
	default 30

	Default for ICMP timeout.

nf_conntrack_icmpv6_timeout - INTEGER (seconds)
	default 30

	Default for ICMP6 timeout.

nf_conntrack_log_invalid - INTEGER
	- 0   - disable (default)
	- 1   - log ICMP packets
	- 6   - log TCP packets
	- 17  - log UDP packets
	- 33  - log DCCP packets
	- 41  - log ICMPv6 packets
	- 136 - log UDPLITE packets
	- 255 - log packets of any protocol

	Log invalid packets of a type specified by value.

nf_conntrack_max - INTEGER
        Maximum number of allowed connection tracking entries. This value is set
        to nf_conntrack_buckets by default.
        Note that connection tracking entries are added to the table twice -- once
        for the original direction and once for the reply direction (i.e., with
        the reversed address). This means that with default settings a maxed-out
        table will have a average hash chain length of 2, not 1.

nf_conntrack_tcp_be_liberal - BOOLEAN
	- 0 - disabled (default)
	- not 0 - enabled

	Be conservative in what you do, be liberal in what you accept from others.
	If it's non-zero, we mark only out of window RST segments as INVALID.

nf_conntrack_tcp_ignore_invalid_rst - BOOLEAN
	- 0 - disabled (default)
	- 1 - enabled

	If it's 1, we don't mark out of window RST segments as INVALID.

nf_conntrack_tcp_loose - BOOLEAN
	- 0 - disabled
	- not 0 - enabled (default)

	If it is set to zero, we disable picking up already established
	connections.

nf_conntrack_tcp_max_retrans - INTEGER
	default 3

	Maximum number of packets that can be retransmitted without
	received an (acceptable) ACK from the destination. If this number
	is reached, a shorter timer will be started.

nf_conntrack_tcp_timeout_close - INTEGER (seconds)
	default 10

nf_conntrack_tcp_timeout_close_wait - INTEGER (seconds)
	default 60

nf_conntrack_tcp_timeout_established - INTEGER (seconds)
	default 432000 (5 days)

nf_conntrack_tcp_timeout_fin_wait - INTEGER (seconds)
	default 120

nf_conntrack_tcp_timeout_last_ack - INTEGER (seconds)
	default 30

nf_conntrack_tcp_timeout_max_retrans - INTEGER (seconds)
	default 300

nf_conntrack_tcp_timeout_syn_recv - INTEGER (seconds)
	default 60

nf_conntrack_tcp_timeout_syn_sent - INTEGER (seconds)
	default 120

nf_conntrack_tcp_timeout_time_wait - INTEGER (seconds)
	default 120

nf_conntrack_tcp_timeout_unacknowledged - INTEGER (seconds)
	default 300

nf_conntrack_timestamp - BOOLEAN
	- 0 - disabled (default)
	- not 0 - enabled

	Enable connection tracking flow timestamping.

nf_conntrack_udp_timeout - INTEGER (seconds)
	default 30

nf_conntrack_udp_timeout_stream - INTEGER (seconds)
	default 120

	This extended timeout will be used in case there is an UDP stream
	detected.

nf_conntrack_gre_timeout - INTEGER (seconds)
	default 30

nf_conntrack_gre_timeout_stream - INTEGER (seconds)
	default 180

	This extended timeout will be used in case there is an GRE stream
	detected.

nf_hooks_lwtunnel - BOOLEAN
	- 0 - disabled (default)
	- not 0 - enabled

	If this option is enabled, the lightweight tunnel netfilter hooks are
	enabled. This option cannot be disabled once it is enabled.

nf_flowtable_tcp_timeout - INTEGER (seconds)
        default 30

        Control offload timeout for tcp connections.
        TCP connections may be offloaded from nf conntrack to nf flow table.
        Once aged, the connection is returned to nf conntrack with tcp pickup timeout.

nf_flowtable_udp_timeout - INTEGER (seconds)
        default 30

        Control offload timeout for udp connections.
        UDP connections may be offloaded from nf conntrack to nf flow table.
        Once aged, the connection is returned to nf conntrack with udp pickup timeout.