code:
.. _kernel_tls:

==========
Kernel TLS
==========

Overview
========

Transport Layer Security (TLS) is a Upper Layer Protocol (ULP) that runs over
TCP. TLS provides end-to-end data integrity and confidentiality.

User interface
==============

Creating a TLS connection
-------------------------

First create a new TCP socket and set the TLS ULP.

.. code-block:: c

  sock = socket(AF_INET, SOCK_STREAM, 0);
  setsockopt(sock, SOL_TCP, TCP_ULP, "tls", sizeof("tls"));

Setting the TLS ULP allows us to set/get TLS socket options. Currently
only the symmetric encryption is handled in the kernel.  After the TLS
handshake is complete, we have all the parameters required to move the
data-path to the kernel. There is a separate socket option for moving
the transmit and the receive into the kernel.

.. code-block:: c

  /* From linux/tls.h */
  struct tls_crypto_info {
          unsigned short version;
          unsigned short cipher_type;
  };

  struct tls12_crypto_info_aes_gcm_128 {
          struct tls_crypto_info info;
          unsigned char iv[TLS_CIPHER_AES_GCM_128_IV_SIZE];
          unsigned char key[TLS_CIPHER_AES_GCM_128_KEY_SIZE];
          unsigned char salt[TLS_CIPHER_AES_GCM_128_SALT_SIZE];
          unsigned char rec_seq[TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE];
  };

  struct tls12_crypto_info_aes_gcm_128 crypto_info;

  crypto_info.info.version = TLS_1_2_VERSION;
  crypto_info.info.cipher_type = TLS_CIPHER_AES_GCM_128;
  memcpy(crypto_info.iv, iv_write, TLS_CIPHER_AES_GCM_128_IV_SIZE);
  memcpy(crypto_info.rec_seq, seq_number_write,
					TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE);
  memcpy(crypto_info.key, cipher_key_write, TLS_CIPHER_AES_GCM_128_KEY_SIZE);
  memcpy(crypto_info.salt, implicit_iv_write, TLS_CIPHER_AES_GCM_128_SALT_SIZE);

  setsockopt(sock, SOL_TLS, TLS_TX, &crypto_info, sizeof(crypto_info));

Transmit and receive are set separately, but the setup is the same, using either
TLS_TX or TLS_RX.

Sending TLS application data
----------------------------

After setting the TLS_TX socket option all application data sent over this
socket is encrypted using TLS and the parameters provided in the socket option.
For example, we can send an encrypted hello world record as follows:

.. code-block:: c

  const char *msg = "hello world\n";
  send(sock, msg, strlen(msg));

send() data is directly encrypted from the userspace buffer provided
to the encrypted kernel send buffer if possible.

The sendfile system call will send the file's data over TLS records of maximum
length (2^14).

.. code-block:: c

  file = open(filename, O_RDONLY);
  fstat(file, &stat);
  sendfile(sock, file, &offset, stat.st_size);

TLS records are created and sent after each send() call, unless
MSG_MORE is passed.  MSG_MORE will delay creation of a record until
MSG_MORE is not passed, or the maximum record size is reached.

The kernel will need to allocate a buffer for the encrypted data.
This buffer is allocated at the time send() is called, such that
either the entire send() call will return -ENOMEM (or block waiting
for memory), or the encryption will always succeed.  If send() returns
-ENOMEM and some data was left on the socket buffer from a previous
call using MSG_MORE, the MSG_MORE data is left on the socket buffer.

Receiving TLS application data
------------------------------

After setting the TLS_RX socket option, all recv family socket calls
are decrypted using TLS parameters provided.  A full TLS record must
be received before decryption can happen.

.. code-block:: c

  char buffer[16384];
  recv(sock, buffer, 16384);

Received data is decrypted directly in to the user buffer if it is
large enough, and no additional allocations occur.  If the userspace
buffer is too small, data is decrypted in the kernel and copied to
userspace.

``EINVAL`` is returned if the TLS version in the received message does not
match the version passed in setsockopt.

``EMSGSIZE`` is returned if the received message is too big.

``EBADMSG`` is returned if decryption failed for any other reason.

Send TLS control messages
-------------------------

Other than application data, TLS has control messages such as alert
messages (record type 21) and handshake messages (record type 22), etc.
These messages can be sent over the socket by providing the TLS record type
via a CMSG. For example the following function sends @data of @length bytes
using a record of type @record_type.

.. code-block:: c

  /* send TLS control message using record_type */
  static int klts_send_ctrl_message(int sock, unsigned char record_type,
                                    void *data, size_t length)
  {
        struct msghdr msg = {0};
        int cmsg_len = sizeof(record_type);
        struct cmsghdr *cmsg;
        char buf[CMSG_SPACE(cmsg_len)];
        struct iovec msg_iov;   /* Vector of data to send/receive into.  */

        msg.msg_control = buf;
        msg.msg_controllen = sizeof(buf);
        cmsg = CMSG_FIRSTHDR(&msg);
        cmsg->cmsg_level = SOL_TLS;
        cmsg->cmsg_type = TLS_SET_RECORD_TYPE;
        cmsg->cmsg_len = CMSG_LEN(cmsg_len);
        *CMSG_DATA(cmsg) = record_type;
        msg.msg_controllen = cmsg->cmsg_len;

        msg_iov.iov_base = data;
        msg_iov.iov_len = length;
        msg.msg_iov = &msg_iov;
        msg.msg_iovlen = 1;

        return sendmsg(sock, &msg, 0);
  }

Control message data should be provided unencrypted, and will be
encrypted by the kernel.

Receiving TLS control messages
------------------------------

TLS control messages are passed in the userspace buffer, with message
type passed via cmsg.  If no cmsg buffer is provided, an error is
returned if a control message is received.  Data messages may be
received without a cmsg buffer set.

.. code-block:: c

  char buffer[16384];
  char cmsg[CMSG_SPACE(sizeof(unsigned char))];
  struct msghdr msg = {0};
  msg.msg_control = cmsg;
  msg.msg_controllen = sizeof(cmsg);

  struct iovec msg_iov;
  msg_iov.iov_base = buffer;
  msg_iov.iov_len = 16384;

  msg.msg_iov = &msg_iov;
  msg.msg_iovlen = 1;

  int ret = recvmsg(sock, &msg, 0 /* flags */);

  struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
  if (cmsg->cmsg_level == SOL_TLS &&
      cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
      int record_type = *((unsigned char *)CMSG_DATA(cmsg));
      // Do something with record_type, and control message data in
      // buffer.
      //
      // Note that record_type may be == to application data (23).
  } else {
      // Buffer contains application data.
  }

recv will never return data from mixed types of TLS records.

Integrating in to userspace TLS library
---------------------------------------

At a high level, the kernel TLS ULP is a replacement for the record
layer of a userspace TLS library.

A patchset to OpenSSL to use ktls as the record layer is
`here <https://github.com/Mellanox/openssl/commits/tls_rx2>`_.

`An example <https://github.com/ktls/af_ktls-tool/commits/RX>`_
of calling send directly after a handshake using gnutls.
Since it doesn't implement a full record layer, control
messages are not supported.

Optional optimizations
----------------------

There are certain condition-specific optimizations the TLS ULP can make,
if requested. Those optimizations are either not universally beneficial
or may impact correctness, hence they require an opt-in.
All options are set per-socket using setsockopt(), and their
state can be checked using getsockopt() and via socket diag (``ss``).

TLS_TX_ZEROCOPY_RO
~~~~~~~~~~~~~~~~~~

For device offload only. Allow sendfile() data to be transmitted directly
to the NIC without making an in-kernel copy. This allows true zero-copy
behavior when device offload is enabled.

The application must make sure that the data is not modified between being
submitted and transmission completing. In other words this is mostly
applicable if the data sent on a socket via sendfile() is read-only.

Modifying the data may result in different versions of the data being used
for the original TCP transmission and TCP retransmissions. To the receiver
this will look like TLS records had been tampered with and will result
in record authentication failures.

TLS_RX_EXPECT_NO_PAD
~~~~~~~~~~~~~~~~~~~~

TLS 1.3 only. Expect the sender to not pad records. This allows the data
to be decrypted directly into user space buffers with TLS 1.3.

This optimization is safe to enable only if the remote end is trusted,
otherwise it is an attack vector to doubling the TLS processing cost.

If the record decrypted turns out to had been padded or is not a data
record it will be decrypted again into a kernel buffer without zero copy.
Such events are counted in the ``TlsDecryptRetry`` statistic.

Statistics
==========

TLS implementation exposes the following per-namespace statistics
(``/proc/net/tls_stat``):

- ``TlsCurrTxSw``, ``TlsCurrRxSw`` -
  number of TX and RX sessions currently installed where host handles
  cryptography

- ``TlsCurrTxDevice``, ``TlsCurrRxDevice`` -
  number of TX and RX sessions currently installed where NIC handles
  cryptography

- ``TlsTxSw``, ``TlsRxSw`` -
  number of TX and RX sessions opened with host cryptography

- ``TlsTxDevice``, ``TlsRxDevice`` -
  number of TX and RX sessions opened with NIC cryptography

- ``TlsDecryptError`` -
  record decryption failed (e.g. due to incorrect authentication tag)

- ``TlsDeviceRxResync`` -
  number of RX resyncs sent to NICs handling cryptography

- ``TlsDecryptRetry`` -
  number of RX records which had to be re-decrypted due to
  ``TLS_RX_EXPECT_NO_PAD`` mis-prediction. Note that this counter will
  also increment for non-data records.

- ``TlsRxNoPadViolation`` -
  number of data RX records which had to be re-decrypted due to
  ``TLS_RX_EXPECT_NO_PAD`` mis-prediction.


code:
.. SPDX-License-Identifier: GPL-2.0

=========================
Transparent proxy support
=========================

This feature adds Linux 2.2-like transparent proxy support to current kernels.
To use it, enable the socket match and the TPROXY target in your kernel config.
You will need policy routing too, so be sure to enable that as well.

From Linux 4.18 transparent proxy support is also available in nf_tables.

1. Making non-local sockets work
================================

The idea is that you identify packets with destination address matching a local
socket on your box, set the packet mark to a certain value::

    # iptables -t mangle -N DIVERT
    # iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    # iptables -t mangle -A DIVERT -j MARK --set-mark 1
    # iptables -t mangle -A DIVERT -j ACCEPT

Alternatively you can do this in nft with the following commands::

    # nft add table filter
    # nft add chain filter divert "{ type filter hook prerouting priority -150; }"
    # nft add rule filter divert meta l4proto tcp socket transparent 1 meta mark set 1 accept

And then match on that value using policy routing to have those packets
delivered locally::

    # ip rule add fwmark 1 lookup 100
    # ip route add local 0.0.0.0/0 dev lo table 100

Because of certain restrictions in the IPv4 routing output code you'll have to
modify your application to allow it to send datagrams _from_ non-local IP
addresses. All you have to do is enable the (SOL_IP, IP_TRANSPARENT) socket
option before calling bind::

    fd = socket(AF_INET, SOCK_STREAM, 0);
    /* - 8< -*/
    int value = 1;
    setsockopt(fd, SOL_IP, IP_TRANSPARENT, &value, sizeof(value));
    /* - 8< -*/
    name.sin_family = AF_INET;
    name.sin_port = htons(0xCAFE);
    name.sin_addr.s_addr = htonl(0xDEADBEEF);
    bind(fd, &name, sizeof(name));

A trivial patch for netcat is available here:
http://people.netfilter.org/hidden/tproxy/netcat-ip_transparent-support.patch

2. Redirecting traffic
======================

Transparent proxying often involves "intercepting" traffic on a router. This is
usually done with the iptables REDIRECT target; however, there are serious
limitations of that method. One of the major issues is that it actually
modifies the packets to change the destination address -- which might not be
acceptable in certain situations. (Think of proxying UDP for example: you won't
be able to find out the original destination address. Even in case of TCP
getting the original destination address is racy.)

The 'TPROXY' target provides similar functionality without relying on NAT. Simply
add rules like this to the iptables ruleset above::

    # iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY \
      --tproxy-mark 0x1/0x1 --on-port 50080

Or the following rule to nft:

# nft add rule filter divert tcp dport 80 tproxy to :50080 meta mark set 1 accept

Note that for this to work you'll have to modify the proxy to enable (SOL_IP,
IP_TRANSPARENT) for the listening socket.

As an example implementation, tcprdr is available here:
https://git.breakpoint.cc/cgit/fw/tcprdr.git/
This tool is written by Florian Westphal and it was used for testing during the
nf_tables implementation.

3. Iptables and nf_tables extensions
====================================

To use tproxy you'll need to have the following modules compiled for iptables:

 - NETFILTER_XT_MATCH_SOCKET
 - NETFILTER_XT_TARGET_TPROXY

Or the floowing modules for nf_tables:

 - NFT_SOCKET
 - NFT_TPROXY

4. Application support
======================

4.1. Squid
----------

Squid 3.HEAD has support built-in. To use it, pass
'--enable-linux-netfilter' to configure and set the 'tproxy' option on
the HTTP listener you redirect traffic to with the TPROXY iptables
target.

For more information please consult the following page on the Squid
wiki: http://wiki.squid-cache.org/Features/Tproxy4

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

===============================
Universal TUN/TAP device driver
===============================

Copyright |copy| 1999-2000 Maxim Krasnyansky <max_mk@yahoo.com>

  Linux, Solaris drivers
  Copyright |copy| 1999-2000 Maxim Krasnyansky <max_mk@yahoo.com>

  FreeBSD TAP driver
  Copyright |copy| 1999-2000 Maksim Yevmenkin <m_evmenkin@yahoo.com>

  Revision of this document 2002 by Florian Thiel <florian.thiel@gmx.net>

1. Description
==============

  TUN/TAP provides packet reception and transmission for user space programs.
  It can be seen as a simple Point-to-Point or Ethernet device, which,
  instead of receiving packets from physical media, receives them from
  user space program and instead of sending packets via physical media
  writes them to the user space program.

  In order to use the driver a program has to open /dev/net/tun and issue a
  corresponding ioctl() to register a network device with the kernel. A network
  device will appear as tunXX or tapXX, depending on the options chosen. When
  the program closes the file descriptor, the network device and all
  corresponding routes will disappear.

  Depending on the type of device chosen the userspace program has to read/write
  IP packets (with tun) or ethernet frames (with tap). Which one is being used
  depends on the flags given with the ioctl().

  The package from http://vtun.sourceforge.net/tun contains two simple examples
  for how to use tun and tap devices. Both programs work like a bridge between
  two network interfaces.
  br_select.c - bridge based on select system call.
  br_sigio.c  - bridge based on async io and SIGIO signal.
  However, the best example is VTun http://vtun.sourceforge.net :))

2. Configuration
================

  Create device node::

     mkdir /dev/net (if it doesn't exist already)
     mknod /dev/net/tun c 10 200

  Set permissions::

     e.g. chmod 0666 /dev/net/tun

  There's no harm in allowing the device to be accessible by non-root users,
  since CAP_NET_ADMIN is required for creating network devices or for
  connecting to network devices which aren't owned by the user in question.
  If you want to create persistent devices and give ownership of them to
  unprivileged users, then you need the /dev/net/tun device to be usable by
  those users.

  Driver module autoloading

     Make sure that "Kernel module loader" - module auto-loading
     support is enabled in your kernel.  The kernel should load it on
     first access.

  Manual loading

     insert the module by hand::

	modprobe tun

  If you do it the latter way, you have to load the module every time you
  need it, if you do it the other way it will be automatically loaded when
  /dev/net/tun is being opened.

3. Program interface
====================

3.1 Network device allocation
-----------------------------

``char *dev`` should be the name of the device with a format string (e.g.
"tun%d"), but (as far as I can see) this can be any valid network device name.
Note that the character pointer becomes overwritten with the real device name
(e.g. "tun0")::

  #include <linux/if.h>
  #include <linux/if_tun.h>

  int tun_alloc(char *dev)
  {
      struct ifreq ifr;
      int fd, err;

      if( (fd = open("/dev/net/tun", O_RDWR)) < 0 )
	 return tun_alloc_old(dev);

      memset(&ifr, 0, sizeof(ifr));

      /* Flags: IFF_TUN   - TUN device (no Ethernet headers)
       *        IFF_TAP   - TAP device
       *
       *        IFF_NO_PI - Do not provide packet information
       */
      ifr.ifr_flags = IFF_TUN;
      if( *dev )
	 strscpy_pad(ifr.ifr_name, dev, IFNAMSIZ);

      if( (err = ioctl(fd, TUNSETIFF, (void *) &ifr)) < 0 ){
	 close(fd);
	 return err;
      }
      strcpy(dev, ifr.ifr_name);
      return fd;
  }

3.2 Frame format
----------------

If flag IFF_NO_PI is not set each frame format is::

     Flags [2 bytes]
     Proto [2 bytes]
     Raw protocol(IP, IPv6, etc) frame.

3.3 Multiqueue tuntap interface
-------------------------------

From version 3.8, Linux supports multiqueue tuntap which can uses multiple
file descriptors (queues) to parallelize packets sending or receiving. The
device allocation is the same as before, and if user wants to create multiple
queues, TUNSETIFF with the same device name must be called many times with
IFF_MULTI_QUEUE flag.

``char *dev`` should be the name of the device, queues is the number of queues
to be created, fds is used to store and return the file descriptors (queues)
created to the caller. Each file descriptor were served as the interface of a
queue which could be accessed by userspace.

::

  #include <linux/if.h>
  #include <linux/if_tun.h>

  int tun_alloc_mq(char *dev, int queues, int *fds)
  {
      struct ifreq ifr;
      int fd, err, i;

      if (!dev)
	  return -1;

      memset(&ifr, 0, sizeof(ifr));
      /* Flags: IFF_TUN   - TUN device (no Ethernet headers)
       *        IFF_TAP   - TAP device
       *
       *        IFF_NO_PI - Do not provide packet information
       *        IFF_MULTI_QUEUE - Create a queue of multiqueue device
       */
      ifr.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_MULTI_QUEUE;
      strcpy(ifr.ifr_name, dev);

      for (i = 0; i < queues; i++) {
	  if ((fd = open("/dev/net/tun", O_RDWR)) < 0)
	     goto err;
	  err = ioctl(fd, TUNSETIFF, (void *)&ifr);
	  if (err) {
	     close(fd);
	     goto err;
	  }
	  fds[i] = fd;
      }

      return 0;
  err:
      for (--i; i >= 0; i--)
	  close(fds[i]);
      return err;
  }

A new ioctl(TUNSETQUEUE) were introduced to enable or disable a queue. When
calling it with IFF_DETACH_QUEUE flag, the queue were disabled. And when
calling it with IFF_ATTACH_QUEUE flag, the queue were enabled. The queue were
enabled by default after it was created through TUNSETIFF.

fd is the file descriptor (queue) that we want to enable or disable, when
enable is true we enable it, otherwise we disable it::

  #include <linux/if.h>
  #include <linux/if_tun.h>

  int tun_set_queue(int fd, int enable)
  {
      struct ifreq ifr;

      memset(&ifr, 0, sizeof(ifr));

      if (enable)
	 ifr.ifr_flags = IFF_ATTACH_QUEUE;
      else
	 ifr.ifr_flags = IFF_DETACH_QUEUE;

      return ioctl(fd, TUNSETQUEUE, (void *)&ifr);
  }

Universal TUN/TAP device driver Frequently Asked Question
=========================================================

1. What platforms are supported by TUN/TAP driver ?

Currently driver has been written for 3 Unices:

  - Linux kernels 2.2.x, 2.4.x
  - FreeBSD 3.x, 4.x, 5.x
  - Solaris 2.6, 7.0, 8.0

2. What is TUN/TAP driver used for?

As mentioned above, main purpose of TUN/TAP driver is tunneling.
It is used by VTun (http://vtun.sourceforge.net).

Another interesting application using TUN/TAP is pipsecd
(http://perso.enst.fr/~beyssac/pipsec/), a userspace IPSec
implementation that can use complete kernel routing (unlike FreeS/WAN).

3. How does Virtual network device actually work ?

Virtual network device can be viewed as a simple Point-to-Point or
Ethernet device, which instead of receiving packets from a physical
media, receives them from user space program and instead of sending
packets via physical media sends them to the user space program.

Let's say that you configured IPv6 on the tap0, then whenever
the kernel sends an IPv6 packet to tap0, it is passed to the application
(VTun for example). The application encrypts, compresses and sends it to
the other side over TCP or UDP. The application on the other side decompresses
and decrypts the data received and writes the packet to the TAP device,
the kernel handles the packet like it came from real physical device.

4. What is the difference between TUN driver and TAP driver?

TUN works with IP frames. TAP works with Ethernet frames.

This means that you have to read/write IP packets when you are using tun and
ethernet frames when using tap.

5. What is the difference between BPF and TUN/TAP driver?

BPF is an advanced packet filter. It can be attached to existing
network interface. It does not provide a virtual network interface.
A TUN/TAP driver does provide a virtual network interface and it is possible
to attach BPF to this interface.

6. Does TAP driver support kernel Ethernet bridging?

Yes. Linux and FreeBSD drivers support Ethernet bridging.


code:
.. SPDX-License-Identifier: GPL-2.0

================================
The UDP-Lite protocol (RFC 3828)
================================

  UDP-Lite is a Standards-Track IETF transport protocol whose characteristic
  is a variable-length checksum. This has advantages for transport of multimedia
  (video, VoIP) over wireless networks, as partly damaged packets can still be
  fed into the codec instead of being discarded due to a failed checksum test.

  This file briefly describes the existing kernel support and the socket API.
  For in-depth information, you can consult:

   - The UDP-Lite Homepage:
     http://web.archive.org/web/%2E/http://www.erg.abdn.ac.uk/users/gerrit/udp-lite/

     From here you can also download some example application source code.

   - The UDP-Lite HOWTO on
     http://web.archive.org/web/%2E/http://www.erg.abdn.ac.uk/users/gerrit/udp-lite/files/UDP-Lite-HOWTO.txt

   - The Wireshark UDP-Lite WiKi (with capture files):
     https://wiki.wireshark.org/Lightweight_User_Datagram_Protocol

   - The Protocol Spec, RFC 3828, http://www.ietf.org/rfc/rfc3828.txt

1. Applications
===============

  Several applications have been ported successfully to UDP-Lite. Ethereal
  (now called wireshark) has UDP-Litev4/v6 support by default.

  Porting applications to UDP-Lite is straightforward: only socket level and
  IPPROTO need to be changed; senders additionally set the checksum coverage
  length (default = header length = 8). Details are in the next section.

2. Programming API
==================

  UDP-Lite provides a connectionless, unreliable datagram service and hence
  uses the same socket type as UDP. In fact, porting from UDP to UDP-Lite is
  very easy: simply add ``IPPROTO_UDPLITE`` as the last argument of the
  socket(2) call so that the statement looks like::

      s = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDPLITE);

  or, respectively,

  ::

      s = socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDPLITE);

  With just the above change you are able to run UDP-Lite services or connect
  to UDP-Lite servers. The kernel will assume that you are not interested in
  using partial checksum coverage and so emulate UDP mode (full coverage).

  To make use of the partial checksum coverage facilities requires setting a
  single socket option, which takes an integer specifying the coverage length:

    * Sender checksum coverage: UDPLITE_SEND_CSCOV

      For example::

	int val = 20;
	setsockopt(s, SOL_UDPLITE, UDPLITE_SEND_CSCOV, &val, sizeof(int));

      sets the checksum coverage length to 20 bytes (12b data + 8b header).
      Of each packet only the first 20 bytes (plus the pseudo-header) will be
      checksummed. This is useful for RTP applications which have a 12-byte
      base header.

    * Receiver checksum coverage: UDPLITE_RECV_CSCOV

      This option is the receiver-side analogue. It is truly optional, i.e. not
      required to enable traffic with partial checksum coverage. Its function is
      that of a traffic filter: when enabled, it instructs the kernel to drop
      all packets which have a coverage _less_ than this value. For example, if
      RTP and UDP headers are to be protected, a receiver can enforce that only
      packets with a minimum coverage of 20 are admitted::

	int min = 20;
	setsockopt(s, SOL_UDPLITE, UDPLITE_RECV_CSCOV, &min, sizeof(int));

  The calls to getsockopt(2) are analogous. Being an extension and not a stand-
  alone protocol, all socket options known from UDP can be used in exactly the
  same manner as before, e.g. UDP_CORK or UDP_ENCAP.

  A detailed discussion of UDP-Lite checksum coverage options is in section IV.

3. Header Files
===============

  The socket API requires support through header files in /usr/include:

    * /usr/include/netinet/in.h
      to define IPPROTO_UDPLITE

    * /usr/include/netinet/udplite.h
      for UDP-Lite header fields and protocol constants

  For testing purposes, the following can serve as a ``mini`` header file::

    #define IPPROTO_UDPLITE       136
    #define SOL_UDPLITE           136
    #define UDPLITE_SEND_CSCOV     10
    #define UDPLITE_RECV_CSCOV     11

  Ready-made header files for various distros are in the UDP-Lite tarball.

4. Kernel Behaviour with Regards to the Various Socket Options
==============================================================

  To enable debugging messages, the log level need to be set to 8, as most
  messages use the KERN_DEBUG level (7).

  1) Sender Socket Options

  If the sender specifies a value of 0 as coverage length, the module
  assumes full coverage, transmits a packet with coverage length of 0
  and according checksum.  If the sender specifies a coverage < 8 and
  different from 0, the kernel assumes 8 as default value.  Finally,
  if the specified coverage length exceeds the packet length, the packet
  length is used instead as coverage length.

  2) Receiver Socket Options

  The receiver specifies the minimum value of the coverage length it
  is willing to accept.  A value of 0 here indicates that the receiver
  always wants the whole of the packet covered. In this case, all
  partially covered packets are dropped and an error is logged.

  It is not possible to specify illegal values (<0 and <8); in these
  cases the default of 8 is assumed.

  All packets arriving with a coverage value less than the specified
  threshold are discarded, these events are also logged.

  3) Disabling the Checksum Computation

  On both sender and receiver, checksumming will always be performed
  and cannot be disabled using SO_NO_CHECK. Thus::

	setsockopt(sockfd, SOL_SOCKET, SO_NO_CHECK,  ... );

  will always will be ignored, while the value of::

	getsockopt(sockfd, SOL_SOCKET, SO_NO_CHECK, &value, ...);

  is meaningless (as in TCP). Packets with a zero checksum field are
  illegal (cf. RFC 3828, sec. 3.1) and will be silently discarded.

  4) Fragmentation

  The checksum computation respects both buffersize and MTU. The size
  of UDP-Lite packets is determined by the size of the send buffer. The
  minimum size of the send buffer is 2048 (defined as SOCK_MIN_SNDBUF
  in include/net/sock.h), the default value is configurable as
  net.core.wmem_default or via setting the SO_SNDBUF socket(7)
  option. The maximum upper bound for the send buffer is determined
  by net.core.wmem_max.

  Given a payload size larger than the send buffer size, UDP-Lite will
  split the payload into several individual packets, filling up the
  send buffer size in each case.

  The precise value also depends on the interface MTU. The interface MTU,
  in turn, may trigger IP fragmentation. In this case, the generated
  UDP-Lite packet is split into several IP packets, of which only the
  first one contains the L4 header.

  The send buffer size has implications on the checksum coverage length.
  Consider the following example::

    Payload: 1536 bytes          Send Buffer:     1024 bytes
    MTU:     1500 bytes          Coverage Length:  856 bytes

  UDP-Lite will ship the 1536 bytes in two separate packets::

    Packet 1: 1024 payload + 8 byte header + 20 byte IP header = 1052 bytes
    Packet 2:  512 payload + 8 byte header + 20 byte IP header =  540 bytes

  The coverage packet covers the UDP-Lite header and 848 bytes of the
  payload in the first packet, the second packet is fully covered. Note
  that for the second packet, the coverage length exceeds the packet
  length. The kernel always re-adjusts the coverage length to the packet
  length in such cases.

  As an example of what happens when one UDP-Lite packet is split into
  several tiny fragments, consider the following example::

    Payload: 1024 bytes            Send buffer size: 1024 bytes
    MTU:      300 bytes            Coverage length:   575 bytes

    +-+-----------+--------------+--------------+--------------+
    |8|    272    |      280     |     280      |     280      |
    +-+-----------+--------------+--------------+--------------+
		280            560            840           1032
					^
    *****checksum coverage*************

  The UDP-Lite module generates one 1032 byte packet (1024 + 8 byte
  header). According to the interface MTU, these are split into 4 IP
  packets (280 byte IP payload + 20 byte IP header). The kernel module
  sums the contents of the entire first two packets, plus 15 bytes of
  the last packet before releasing the fragments to the IP module.

  To see the analogous case for IPv6 fragmentation, consider a link
  MTU of 1280 bytes and a write buffer of 3356 bytes. If the checksum
  coverage is less than 1232 bytes (MTU minus IPv6/fragment header
  lengths), only the first fragment needs to be considered. When using
  larger checksum coverage lengths, each eligible fragment needs to be
  checksummed. Suppose we have a checksum coverage of 3062. The buffer
  of 3356 bytes will be split into the following fragments::

    Fragment 1: 1280 bytes carrying  1232 bytes of UDP-Lite data
    Fragment 2: 1280 bytes carrying  1232 bytes of UDP-Lite data
    Fragment 3:  948 bytes carrying   900 bytes of UDP-Lite data

  The first two fragments have to be checksummed in full, of the last
  fragment only 598 (= 3062 - 2*1232) bytes are checksummed.

  While it is important that such cases are dealt with correctly, they
  are (annoyingly) rare: UDP-Lite is designed for optimising multimedia
  performance over wireless (or generally noisy) links and thus smaller
  coverage lengths are likely to be expected.

5. UDP-Lite Runtime Statistics and their Meaning
================================================

  Exceptional and error conditions are logged to syslog at the KERN_DEBUG
  level.  Live statistics about UDP-Lite are available in /proc/net/snmp
  and can (with newer versions of netstat) be viewed using::

			    netstat -svu

  This displays UDP-Lite statistics variables, whose meaning is as follows.

   ============     =====================================================
   InDatagrams      The total number of datagrams delivered to users.

   NoPorts          Number of packets received to an unknown port.
		    These cases are counted separately (not as InErrors).

   InErrors         Number of erroneous UDP-Lite packets. Errors include:

		      * internal socket queue receive errors
		      * packet too short (less than 8 bytes or stated
			coverage length exceeds received length)
		      * xfrm4_policy_check() returned with error
		      * application has specified larger min. coverage
			length than that of incoming packet
		      * checksum coverage violated
		      * bad checksum

   OutDatagrams     Total number of sent datagrams.
   ============     =====================================================

   These statistics derive from the UDP MIB (RFC 2013).

6. IPtables
===========

  There is packet match support for UDP-Lite as well as support for the LOG target.
  If you copy and paste the following line into /etc/protocols::

    udplite 136     UDP-Lite        # UDP-Lite [RFC 3828]

  then::

	      iptables -A INPUT -p udplite -j LOG

  will produce logging output to syslog. Dropping and rejecting packets also works.

7. Maintainer Address
=====================

  The UDP-Lite patch was developed at

		    University of Aberdeen
		    Electronics Research Group
		    Department of Engineering
		    Fraser Noble Building
		    Aberdeen AB24 3UE; UK

  The current maintainer is Gerrit Renker, <gerrit@erg.abdn.ac.uk>. Initial
  code was developed by William  Stanislaus, <william@erg.abdn.ac.uk>.


code:
.. SPDX-License-Identifier: GPL-2.0

====================================
Virtual Routing and Forwarding (VRF)
====================================

The VRF Device
==============

The VRF device combined with ip rules provides the ability to create virtual
routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
Linux network stack. One use case is the multi-tenancy problem where each
tenant has their own unique routing tables and in the very least need
different default gateways.

Processes can be "VRF aware" by binding a socket to the VRF device. Packets
through the socket then use the routing table associated with the VRF
device. An important feature of the VRF device implementation is that it
impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
(ie., they do not need to be run in each VRF). The design also allows
the use of higher priority ip rules (Policy Based Routing, PBR) to take
precedence over the VRF device rules directing specific traffic as desired.

In addition, VRF devices allow VRFs to be nested within namespaces. For
example network namespaces provide separation of network interfaces at the
device layer, VLANs on the interfaces within a namespace provide L2 separation
and then VRF devices provide L3 separation.

Design
------
A VRF device is created with an associated route table. Network interfaces
are then enslaved to a VRF device::

	 +-----------------------------+
	 |           vrf-blue          |  ===> route table 10
	 +-----------------------------+
	    |        |            |
	 +------+ +------+     +-------------+
	 | eth1 | | eth2 | ... |    bond1    |
	 +------+ +------+     +-------------+
				  |       |
			      +------+ +------+
			      | eth8 | | eth9 |
			      +------+ +------+

Packets received on an enslaved device and are switched to the VRF device
in the IPv4 and IPv6 processing stacks giving the impression that packets
flow through the VRF device. Similarly on egress routing rules are used to
send packets to the VRF device driver before getting sent out the actual
interface. This allows tcpdump on a VRF device to capture all packets into
and out of the VRF as a whole\ [1]_. Similarly, netfilter\ [2]_ and tc rules
can be applied using the VRF device to specify rules that apply to the VRF
domain as a whole.

.. [1] Packets in the forwarded state do not flow through the device, so those
       packets are not seen by tcpdump. Will revisit this limitation in a
       future release.

.. [2] Iptables on ingress supports PREROUTING with skb->dev set to the real
       ingress device and both INPUT and PREROUTING rules with skb->dev set to
       the VRF device. For egress POSTROUTING and OUTPUT rules can be written
       using either the VRF device or real egress device.

Setup
-----
1. VRF device is created with an association to a FIB table.
   e.g,::

	ip link add vrf-blue type vrf table 10
	ip link set dev vrf-blue up

2. An l3mdev FIB rule directs lookups to the table associated with the device.
   A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
   l3mdev rule for IPv4 and IPv6 when the first device is created with a
   default preference of 1000. Users may delete the rule if desired and add
   with a different priority or install per-VRF rules.

   Prior to the v4.8 kernel iif and oif rules are needed for each VRF device::

       ip ru add oif vrf-blue table 10
       ip ru add iif vrf-blue table 10

3. Set the default route for the table (and hence default route for the VRF)::

       ip route add table 10 unreachable default metric 4278198272

   This high metric value ensures that the default unreachable route can
   be overridden by a routing protocol suite.  FRRouting interprets
   kernel metrics as a combined admin distance (upper byte) and priority
   (lower 3 bytes).  Thus the above metric translates to [255/8192].

4. Enslave L3 interfaces to a VRF device::

       ip link set dev eth1 master vrf-blue

   Local and connected routes for enslaved devices are automatically moved to
   the table associated with VRF device. Any additional routes depending on
   the enslaved device are dropped and will need to be reinserted to the VRF
   FIB table following the enslavement.

   The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
   addresses as VRF enslavement changes::

       sysctl -w net.ipv6.conf.all.keep_addr_on_down=1

5. Additional VRF routes are added to associated table::

       ip route add table 10 ...

Applications
------------
Applications that are to work within a VRF need to bind their socket to the
VRF device::

    setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);

or to specify the output device using cmsg and IP_PKTINFO.

By default the scope of the port bindings for unbound sockets is
limited to the default VRF. That is, it will not be matched by packets
arriving on interfaces enslaved to an l3mdev and processes may bind to
the same port if they bind to an l3mdev.

TCP & UDP services running in the default VRF context (ie., not bound
to any VRF device) can work across all VRF domains by enabling the
tcp_l3mdev_accept and udp_l3mdev_accept sysctl options::

    sysctl -w net.ipv4.tcp_l3mdev_accept=1
    sysctl -w net.ipv4.udp_l3mdev_accept=1

These options are disabled by default so that a socket in a VRF is only
selected for packets in that VRF. There is a similar option for RAW
sockets, which is enabled by default for reasons of backwards compatibility.
This is so as to specify the output device with cmsg and IP_PKTINFO, but
using a socket not bound to the corresponding VRF. This allows e.g. older ping
implementations to be run with specifying the device but without executing it
in the VRF. This option can be disabled so that packets received in a VRF
context are only handled by a raw socket bound to the VRF, and packets in the
default VRF are only handled by a socket not bound to any VRF::

    sysctl -w net.ipv4.raw_l3mdev_accept=0

netfilter rules on the VRF device can be used to limit access to services
running in the default VRF context as well.

Using VRF-aware applications (applications which simultaneously create sockets
outside and inside VRFs) in conjunction with ``net.ipv4.tcp_l3mdev_accept=1``
is possible but may lead to problems in some situations. With that sysctl
value, it is unspecified which listening socket will be selected to handle
connections for VRF traffic; ie. either a socket bound to the VRF or an unbound
socket may be used to accept new connections from a VRF. This somewhat
unexpected behavior can lead to problems if sockets are configured with extra
options (ex. TCP MD5 keys) with the expectation that VRF traffic will
exclusively be handled by sockets bound to VRFs, as would be the case with
``net.ipv4.tcp_l3mdev_accept=0``. Finally and as a reminder, regardless of
which listening socket is selected, established sockets will be created in the
VRF based on the ingress interface, as documented earlier.

--------------------------------------------------------------------------------

Using iproute2 for VRFs
=======================
iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
section lists both commands where appropriate -- with the vrf keyword and the
older form without it.

1. Create a VRF

   To instantiate a VRF device and associate it with a table::

       $ ip link add dev NAME type vrf table ID

   As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
   covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
   device create.

2. List VRFs

   To list VRFs that have been created::

       $ ip [-d] link show type vrf
	 NOTE: The -d option is needed to show the table id

   For example::

       $ ip -d link show type vrf
       11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	   link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
	   vrf table 1 addrgenmode eui64
       12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	   link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
	   vrf table 10 addrgenmode eui64
       13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	   link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
	   vrf table 66 addrgenmode eui64
       14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	   link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
	   vrf table 81 addrgenmode eui64

   Or in brief output::

       $ ip -br link show type vrf
       mgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
       red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
       blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
       green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>

3. Assign a Network Interface to a VRF

   Network interfaces are assigned to a VRF by enslaving the netdevice to a
   VRF device::

       $ ip link set dev NAME master NAME

   On enslavement connected and local routes are automatically moved to the
   table associated with the VRF device.

   For example::

       $ ip link set dev eth0 master mgmt

4. Show Devices Assigned to a VRF

   To show devices that have been assigned to a specific VRF add the master
   option to the ip command::

       $ ip link show vrf NAME
       $ ip link show master NAME

   For example::

       $ ip link show vrf red
       3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
	   link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
       4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
	   link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
       7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
	   link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

   Or using the brief output::

       $ ip -br link show vrf red
       eth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
       eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
       eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>

5. Show Neighbor Entries for a VRF

   To list neighbor entries associated with devices enslaved to a VRF device
   add the master option to the ip command::

       $ ip [-6] neigh show vrf NAME
       $ ip [-6] neigh show master NAME

   For example::

       $  ip neigh show vrf red
       10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
       10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE

       $ ip -6 neigh show vrf red
       2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE

6. Show Addresses for a VRF

   To show addresses for interfaces associated with a VRF add the master
   option to the ip command::

       $ ip addr show vrf NAME
       $ ip addr show master NAME

   For example::

	$ ip addr show vrf red
	3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
	    link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
	    inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
	       valid_lft forever preferred_lft forever
	    inet6 2002:1::2/120 scope global
	       valid_lft forever preferred_lft forever
	    inet6 fe80::ff:fe00:202/64 scope link
	       valid_lft forever preferred_lft forever
	4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
	    link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
	    inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
	       valid_lft forever preferred_lft forever
	    inet6 2002:2::2/120 scope global
	       valid_lft forever preferred_lft forever
	    inet6 fe80::ff:fe00:203/64 scope link
	       valid_lft forever preferred_lft forever
	7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
	    link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

   Or in brief format::

	$ ip -br addr show vrf red
	eth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
	eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
	eth5             DOWN

7. Show Routes for a VRF

   To show routes for a VRF use the ip command to display the table associated
   with the VRF device::

       $ ip [-6] route show vrf NAME
       $ ip [-6] route show table ID

   For example::

	$ ip route show vrf red
	unreachable default  metric 4278198272
	broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.2
	10.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2
	local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2
	broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2
	broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.2
	10.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2
	local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2
	broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2

	$ ip -6 route show vrf red
	local 2002:1:: dev lo  proto none  metric 0  pref medium
	local 2002:1::2 dev lo  proto none  metric 0  pref medium
	2002:1::/120 dev eth1  proto kernel  metric 256  pref medium
	local 2002:2:: dev lo  proto none  metric 0  pref medium
	local 2002:2::2 dev lo  proto none  metric 0  pref medium
	2002:2::/120 dev eth2  proto kernel  metric 256  pref medium
	local fe80:: dev lo  proto none  metric 0  pref medium
	local fe80:: dev lo  proto none  metric 0  pref medium
	local fe80::ff:fe00:202 dev lo  proto none  metric 0  pref medium
	local fe80::ff:fe00:203 dev lo  proto none  metric 0  pref medium
	fe80::/64 dev eth1  proto kernel  metric 256  pref medium
	fe80::/64 dev eth2  proto kernel  metric 256  pref medium
	ff00::/8 dev red  metric 256  pref medium
	ff00::/8 dev eth1  metric 256  pref medium
	ff00::/8 dev eth2  metric 256  pref medium
	unreachable default dev lo  metric 4278198272  error -101 pref medium

8. Route Lookup for a VRF

   A test route lookup can be done for a VRF::

       $ ip [-6] route get vrf NAME ADDRESS
       $ ip [-6] route get oif NAME ADDRESS

   For example::

	$ ip route get 10.2.1.40 vrf red
	10.2.1.40 dev eth1  table red  src 10.2.1.2
	    cache

	$ ip -6 route get 2002:1::32 vrf red
	2002:1::32 from :: dev eth1  table red  proto kernel  src 2002:1::2  metric 256  pref medium

9. Removing Network Interface from a VRF

   Network interfaces are removed from a VRF by breaking the enslavement to
   the VRF device::

       $ ip link set dev NAME nomaster

   Connected routes are moved back to the default table and local entries are
   moved to the local table.

   For example::

    $ ip link set dev eth0 nomaster

--------------------------------------------------------------------------------

Commands used in this example::

     cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
     1  mgmt
     10 red
     66 blue
     81 green
     EOF

     function vrf_create
     {
	 VRF=$1
	 TBID=$2

	 # create VRF device
	 ip link add ${VRF} type vrf table ${TBID}

	 if [ "${VRF}" != "mgmt" ]; then
	     ip route add table ${TBID} unreachable default metric 4278198272
	 fi
	 ip link set dev ${VRF} up
     }

     vrf_create mgmt 1
     ip link set dev eth0 master mgmt

     vrf_create red 10
     ip link set dev eth1 master red
     ip link set dev eth2 master red
     ip link set dev eth5 master red

     vrf_create blue 66
     ip link set dev eth3 master blue

     vrf_create green 81
     ip link set dev eth4 master green

     Interface addresses from /etc/network/interfaces:
     auto eth0
     iface eth0 inet static
	   address 10.0.0.2
	   netmask 255.255.255.0
	   gateway 10.0.0.254

     iface eth0 inet6 static
	   address 2000:1::2
	   netmask 120

     auto eth1
     iface eth1 inet static
	   address 10.2.1.2
	   netmask 255.255.255.0

     iface eth1 inet6 static
	   address 2002:1::2
	   netmask 120

     auto eth2
     iface eth2 inet static
	   address 10.2.2.2
	   netmask 255.255.255.0

     iface eth2 inet6 static
	   address 2002:2::2
	   netmask 120

     auto eth3
     iface eth3 inet static
	   address 10.2.3.2
	   netmask 255.255.255.0

     iface eth3 inet6 static
	   address 2002:3::2
	   netmask 120

     auto eth4
     iface eth4 inet static
	   address 10.2.4.2
	   netmask 255.255.255.0

     iface eth4 inet6 static
	   address 2002:4::2
	   netmask 120


code:
.. SPDX-License-Identifier: GPL-2.0

======================================================
Virtual eXtensible Local Area Networking documentation
======================================================

The VXLAN protocol is a tunnelling protocol designed to solve the
problem of limited VLAN IDs (4096) in IEEE 802.1q.  With VXLAN the
size of the identifier is expanded to 24 bits (16777216).

VXLAN is described by IETF RFC 7348, and has been implemented by a
number of vendors.  The protocol runs over UDP using a single
destination port.  This document describes the Linux kernel tunnel
device, there is also a separate implementation of VXLAN for
Openvswitch.

Unlike most tunnels, a VXLAN is a 1 to N network, not just point to
point. A VXLAN device can learn the IP address of the other endpoint
either dynamically in a manner similar to a learning bridge, or make
use of statically-configured forwarding entries.

The management of vxlan is done in a manner similar to its two closest
neighbors GRE and VLAN. Configuring VXLAN requires the version of
iproute2 that matches the kernel release where VXLAN was first merged
upstream.

1. Create vxlan device::

    # ip link add vxlan0 type vxlan id 42 group 239.1.1.1 dev eth1 dstport 4789

This creates a new device named vxlan0.  The device uses the multicast
group 239.1.1.1 over eth1 to handle traffic for which there is no
entry in the forwarding table.  The destination port number is set to
the IANA-assigned value of 4789.  The Linux implementation of VXLAN
pre-dates the IANA's selection of a standard destination port number
and uses the Linux-selected value by default to maintain backwards
compatibility.

2. Delete vxlan device::

    # ip link delete vxlan0

3. Show vxlan info::

    # ip -d link show vxlan0

It is possible to create, destroy and display the vxlan
forwarding table using the new bridge command.

1. Create forwarding table entry::

    # bridge fdb add to 00:17:42:8a:b4:05 dst 192.19.0.2 dev vxlan0

2. Delete forwarding table entry::

    # bridge fdb delete 00:17:42:8a:b4:05 dev vxlan0

3. Show forwarding table::

    # bridge fdb show dev vxlan0

The following NIC features may indicate support for UDP tunnel-related
offloads (most commonly VXLAN features, but support for a particular
encapsulation protocol is NIC specific):

 - `tx-udp_tnl-segmentation`
 - `tx-udp_tnl-csum-segmentation`
    ability to perform TCP segmentation offload of UDP encapsulated frames

 - `rx-udp_tunnel-port-offload`
    receive side parsing of UDP encapsulated frames which allows NICs to
    perform protocol-aware offloads, like checksum validation offload of
    inner frames (only needed by NICs without protocol-agnostic offloads)

For devices supporting `rx-udp_tunnel-port-offload` the list of currently
offloaded ports can be interrogated with `ethtool`::

  $ ethtool --show-tunnels eth0
  Tunnel information for eth0:
    UDP port table 0:
      Size: 4
      Types: vxlan
      No entries
    UDP port table 1:
      Size: 4
      Types: geneve, vxlan-gpe
      Entries (1):
          port 1230, vxlan-gpe


code:
.. SPDX-License-Identifier: GPL-2.0

============================-
X.25 Device Driver Interface
============================-

Version 1.1

			   Jonathan Naylor 26.12.96

This is a description of the messages to be passed between the X.25 Packet
Layer and the X.25 device driver. They are designed to allow for the easy
setting of the LAPB mode from within the Packet Layer.

The X.25 device driver will be coded normally as per the Linux device driver
standards. Most X.25 device drivers will be moderately similar to the
already existing Ethernet device drivers. However unlike those drivers, the
X.25 device driver has a state associated with it, and this information
needs to be passed to and from the Packet Layer for proper operation.

All messages are held in sk_buff's just like real data to be transmitted
over the LAPB link. The first byte of the skbuff indicates the meaning of
the rest of the skbuff, if any more information does exist.

Packet Layer to Device Driver
-----------------------------

First Byte = 0x00 (X25_IFACE_DATA)

This indicates that the rest of the skbuff contains data to be transmitted
over the LAPB link. The LAPB link should already exist before any data is
passed down.

First Byte = 0x01 (X25_IFACE_CONNECT)

Establish the LAPB link. If the link is already established then the connect
confirmation message should be returned as soon as possible.

First Byte = 0x02 (X25_IFACE_DISCONNECT)

Terminate the LAPB link. If it is already disconnected then the disconnect
confirmation message should be returned as soon as possible.

First Byte = 0x03 (X25_IFACE_PARAMS)

LAPB parameters. To be defined.

Device Driver to Packet Layer
-----------------------------

First Byte = 0x00 (X25_IFACE_DATA)

This indicates that the rest of the skbuff contains data that has been
received over the LAPB link.

First Byte = 0x01 (X25_IFACE_CONNECT)

LAPB link has been established. The same message is used for both a LAPB
link connect_confirmation and a connect_indication.

First Byte = 0x02 (X25_IFACE_DISCONNECT)

LAPB link has been terminated. This same message is used for both a LAPB
link disconnect_confirmation and a disconnect_indication.

First Byte = 0x03 (X25_IFACE_PARAMS)

LAPB parameters. To be defined.

Requirements for the device driver
----------------------------------

Packets should not be reordered or dropped when delivering between the
Packet Layer and the device driver.

To avoid packets from being reordered or dropped when delivering from
the device driver to the Packet Layer, the device driver should not
call "netif_rx" to deliver the received packets. Instead, it should
call "netif_receive_skb_core" from softirq context to deliver them.


code:
.. SPDX-License-Identifier: GPL-2.0

==================
Linux X.25 Project
==================

As my third year dissertation at University I have taken it upon myself to
write an X.25 implementation for Linux. My aim is to provide a complete X.25
Packet Layer and a LAPB module to allow for "normal" X.25 to be run using
Linux. There are two sorts of X.25 cards available, intelligent ones that
implement LAPB on the card itself, and unintelligent ones that simply do
framing, bit-stuffing and checksumming. These both need to be handled by the
system.

I therefore decided to write the implementation such that as far as the
Packet Layer is concerned, the link layer was being performed by a lower
layer of the Linux kernel and therefore it did not concern itself with
implementation of LAPB. Therefore the LAPB modules would be called by
unintelligent X.25 card drivers and not by intelligent ones, this would
provide a uniform device driver interface, and simplify configuration.

To confuse matters a little, an 802.2 LLC implementation is also possible
which could allow X.25 to be run over an Ethernet (or Token Ring) and
conform with the JNT "Pink Book", this would have a different interface to
the Packet Layer but there would be no confusion since the class of device
being served by the LLC would be completely separate from LAPB.

Just when you thought that it could not become more confusing, another
option appeared, XOT. This allows X.25 Packet Layer frames to operate over
the Internet using TCP/IP as a reliable link layer. RFC1613 specifies the
format and behaviour of the protocol. If time permits this option will also
be actively considered.

A linux-x25 mailing list has been created at vger.kernel.org to support the
development and use of Linux X.25. It is early days yet, but interested
parties are welcome to subscribe to it. Just send a message to
majordomo@vger.kernel.org with the following in the message body:

subscribe linux-x25
end

The contents of the Subject line are ignored.

Jonathan

g4klx@g4klx.demon.co.uk


code:
.. SPDX-License-Identifier: GPL-2.0

===============================================
XFRM device - offloading the IPsec computations
===============================================

Shannon Nelson <shannon.nelson@oracle.com>
Leon Romanovsky <leonro@nvidia.com>

Overview
========

IPsec is a useful feature for securing network traffic, but the
computational cost is high: a 10Gbps link can easily be brought down
to under 1Gbps, depending on the traffic and link configuration.
Luckily, there are NICs that offer a hardware based IPsec offload which
can radically increase throughput and decrease CPU utilization.  The XFRM
Device interface allows NIC drivers to offer to the stack access to the
hardware offload.

Right now, there are two types of hardware offload that kernel supports.
 * IPsec crypto offload:
   * NIC performs encrypt/decrypt
   * Kernel does everything else
 * IPsec packet offload:
   * NIC performs encrypt/decrypt
   * NIC does encapsulation
   * Kernel and NIC have SA and policy in-sync
   * NIC handles the SA and policies states
   * The Kernel talks to the keymanager

Userland access to the offload is typically through a system such as
libreswan or KAME/raccoon, but the iproute2 'ip xfrm' command set can
be handy when experimenting.  An example command might look something
like this for crypto offload:

  ip x s add proto esp dst 14.0.0.70 src 14.0.0.52 spi 0x07 mode transport \
     reqid 0x07 replay-window 32 \
     aead 'rfc4106(gcm(aes))' 0x44434241343332312423222114131211f4f3f2f1 128 \
     sel src 14.0.0.52/24 dst 14.0.0.70/24 proto tcp \
     offload dev eth4 dir in

and for packet offload

  ip x s add proto esp dst 14.0.0.70 src 14.0.0.52 spi 0x07 mode transport \
     reqid 0x07 replay-window 32 \
     aead 'rfc4106(gcm(aes))' 0x44434241343332312423222114131211f4f3f2f1 128 \
     sel src 14.0.0.52/24 dst 14.0.0.70/24 proto tcp \
     offload packet dev eth4 dir in

  ip x p add src 14.0.0.70 dst 14.0.0.52 offload packet dev eth4 dir in
  tmpl src 14.0.0.70 dst 14.0.0.52 proto esp reqid 10000 mode transport

Yes, that's ugly, but that's what shell scripts and/or libreswan are for.

Callbacks to implement
======================

::

  /* from include/linux/netdevice.h */
  struct xfrmdev_ops {
        /* Crypto and Packet offload callbacks */
	int	(*xdo_dev_state_add) (struct xfrm_state *x);
	void	(*xdo_dev_state_delete) (struct xfrm_state *x);
	void	(*xdo_dev_state_free) (struct xfrm_state *x);
	bool	(*xdo_dev_offload_ok) (struct sk_buff *skb,
				       struct xfrm_state *x);
	void    (*xdo_dev_state_advance_esn) (struct xfrm_state *x);

        /* Solely packet offload callbacks */
	void    (*xdo_dev_state_update_curlft) (struct xfrm_state *x);
	int	(*xdo_dev_policy_add) (struct xfrm_policy *x);
	void	(*xdo_dev_policy_delete) (struct xfrm_policy *x);
	void	(*xdo_dev_policy_free) (struct xfrm_policy *x);
  };

The NIC driver offering ipsec offload will need to implement callbacks
relevant to supported offload to make the offload available to the network
stack's XFRM subsystem. Additionally, the feature bits NETIF_F_HW_ESP and
NETIF_F_HW_ESP_TX_CSUM will signal the availability of the offload.

Flow
====

At probe time and before the call to register_netdev(), the driver should
set up local data structures and XFRM callbacks, and set the feature bits.
The XFRM code's listener will finish the setup on NETDEV_REGISTER.

::

		adapter->netdev->xfrmdev_ops = &ixgbe_xfrmdev_ops;
		adapter->netdev->features |= NETIF_F_HW_ESP;
		adapter->netdev->hw_enc_features |= NETIF_F_HW_ESP;

When new SAs are set up with a request for "offload" feature, the
driver's xdo_dev_state_add() will be given the new SA to be offloaded
and an indication of whether it is for Rx or Tx.  The driver should

	- verify the algorithm is supported for offloads
	- store the SA information (key, salt, target-ip, protocol, etc)
	- enable the HW offload of the SA
	- return status value:

		===========   ===================================
		0             success
		-EOPNETSUPP   offload not supported, try SW IPsec,
                              not applicable for packet offload mode
		other         fail the request
		===========   ===================================

The driver can also set an offload_handle in the SA, an opaque void pointer
that can be used to convey context into the fast-path offload requests::

		xs->xso.offload_handle = context;

When the network stack is preparing an IPsec packet for an SA that has
been setup for offload, it first calls into xdo_dev_offload_ok() with
the skb and the intended offload state to ask the driver if the offload
will serviceable.  This can check the packet information to be sure the
offload can be supported (e.g. IPv4 or IPv6, no IPv4 options, etc) and
return true of false to signify its support.

Crypto offload mode:
When ready to send, the driver needs to inspect the Tx packet for the
offload information, including the opaque context, and set up the packet
send accordingly::

		xs = xfrm_input_state(skb);
		context = xs->xso.offload_handle;
		set up HW for send

The stack has already inserted the appropriate IPsec headers in the
packet data, the offload just needs to do the encryption and fix up the
header values.

When a packet is received and the HW has indicated that it offloaded a
decryption, the driver needs to add a reference to the decoded SA into
the packet's skb.  At this point the data should be decrypted but the
IPsec headers are still in the packet data; they are removed later up
the stack in xfrm_input().

	find and hold the SA that was used to the Rx skb::

		get spi, protocol, and destination IP from packet headers
		xs = find xs from (spi, protocol, dest_IP)
		xfrm_state_hold(xs);

	store the state information into the skb::

		sp = secpath_set(skb);
		if (!sp) return;
		sp->xvec[sp->len++] = xs;
		sp->olen++;

	indicate the success and/or error status of the offload::

		xo = xfrm_offload(skb);
		xo->flags = CRYPTO_DONE;
		xo->status = crypto_status;

	hand the packet to napi_gro_receive() as usual

In ESN mode, xdo_dev_state_advance_esn() is called from xfrm_replay_advance_esn().
Driver will check packet seq number and update HW ESN state machine if needed.

Packet offload mode:
HW adds and deletes XFRM headers. So in RX path, XFRM stack is bypassed if HW
reported success. In TX path, the packet lefts kernel without extra header
and not encrypted, the HW is responsible to perform it.

When the SA is removed by the user, the driver's xdo_dev_state_delete()
and xdo_dev_policy_delete() are asked to disable the offload.  Later,
xdo_dev_state_free() and xdo_dev_policy_free() are called from a garbage
collection routine after all reference counts to the state and policy
have been removed and any remaining resources can be cleared for the
offload state.  How these are used by the driver will depend on specific
hardware needs.

As a netdev is set to DOWN the XFRM stack's netdev listener will call
xdo_dev_state_delete(), xdo_dev_policy_delete(), xdo_dev_state_free() and
xdo_dev_policy_free() on any remaining offloaded states.

Outcome of HW handling packets, the XFRM core can't count hard, soft limits.
The HW/driver are responsible to perform it and provide accurate data when
xdo_dev_state_update_curlft() is called. In case of one of these limits
occuried, the driver needs to call to xfrm_state_check_expire() to make sure
that XFRM performs rekeying sequence.


code:
.. SPDX-License-Identifier: GPL-2.0

==================================
XFRM proc - /proc/net/xfrm_* files
==================================

Masahide NAKAMURA <nakam@linux-ipv6.org>

Transformation Statistics
-------------------------

The xfrm_proc code is a set of statistics showing numbers of packets
dropped by the transformation code and why.  These counters are defined
as part of the linux private MIB.  These counters can be viewed in
/proc/net/xfrm_stat.

Inbound errors
~~~~~~~~~~~~~~

XfrmInError:
	All errors which is not matched others

XfrmInBufferError:
	No buffer is left

XfrmInHdrError:
	Header error

XfrmInNoStates:
	No state is found
	i.e. Either inbound SPI, address, or IPsec protocol at SA is wrong

XfrmInStateProtoError:
	Transformation protocol specific error
	e.g. SA key is wrong

XfrmInStateModeError:
	Transformation mode specific error

XfrmInStateSeqError:
	Sequence error
	i.e. Sequence number is out of window

XfrmInStateExpired:
	State is expired

XfrmInStateMismatch:
	State has mismatch option
	e.g. UDP encapsulation type is mismatch

XfrmInStateInvalid:
	State is invalid

XfrmInTmplMismatch:
	No matching template for states
	e.g. Inbound SAs are correct but SP rule is wrong

XfrmInNoPols:
	No policy is found for states
	e.g. Inbound SAs are correct but no SP is found

XfrmInPolBlock:
	Policy discards

XfrmInPolError:
	Policy error

XfrmAcquireError:
	State hasn't been fully acquired before use

XfrmFwdHdrError:
	Forward routing of a packet is not allowed

Outbound errors
~~~~~~~~~~~~~~~
XfrmOutError:
	All errors which is not matched others

XfrmOutBundleGenError:
	Bundle generation error

XfrmOutBundleCheckError:
	Bundle check error

XfrmOutNoStates:
	No state is found

XfrmOutStateProtoError:
	Transformation protocol specific error

XfrmOutStateModeError:
	Transformation mode specific error

XfrmOutStateSeqError:
	Sequence error
	i.e. Sequence number overflow

XfrmOutStateExpired:
	State is expired

XfrmOutPolBlock:
	Policy discards

XfrmOutPolDead:
	Policy is dead

XfrmOutPolError:
	Policy error

XfrmOutStateInvalid:
	State is invalid, perhaps expired


code:
.. SPDX-License-Identifier: GPL-2.0

====
XFRM
====

The sync patches work is based on initial patches from
Krisztian <hidden@balabit.hu> and others and additional patches
from Jamal <hadi@cyberus.ca>.

The end goal for syncing is to be able to insert attributes + generate
events so that the SA can be safely moved from one machine to another
for HA purposes.
The idea is to synchronize the SA so that the takeover machine can do
the processing of the SA as accurate as possible if it has access to it.

We already have the ability to generate SA add/del/upd events.
These patches add ability to sync and have accurate lifetime byte (to
ensure proper decay of SAs) and replay counters to avoid replay attacks
with as minimal loss at failover time.
This way a backup stays as closely up-to-date as an active member.

Because the above items change for every packet the SA receives,
it is possible for a lot of the events to be generated.
For this reason, we also add a nagle-like algorithm to restrict
the events. i.e we are going to set thresholds to say "let me
know if the replay sequence threshold is reached or 10 secs have passed"
These thresholds are set system-wide via sysctls or can be updated
per SA.

The identified items that need to be synchronized are:
- the lifetime byte counter
note that: lifetime time limit is not important if you assume the failover
machine is known ahead of time since the decay of the time countdown
is not driven by packet arrival.
- the replay sequence for both inbound and outbound

1) Message Structure
----------------------

nlmsghdr:aevent_id:optional-TLVs.

The netlink message types are:

XFRM_MSG_NEWAE and XFRM_MSG_GETAE.

A XFRM_MSG_GETAE does not have TLVs.

A XFRM_MSG_NEWAE will have at least two TLVs (as is
discussed further below).

aevent_id structure looks like::

   struct xfrm_aevent_id {
	     struct xfrm_usersa_id           sa_id;
	     xfrm_address_t                  saddr;
	     __u32                           flags;
	     __u32                           reqid;
   };

The unique SA is identified by the combination of xfrm_usersa_id,
reqid and saddr.

flags are used to indicate different things. The possible
flags are::

	XFRM_AE_RTHR=1, /* replay threshold*/
	XFRM_AE_RVAL=2, /* replay value */
	XFRM_AE_LVAL=4, /* lifetime value */
	XFRM_AE_ETHR=8, /* expiry timer threshold */
	XFRM_AE_CR=16, /* Event cause is replay update */
	XFRM_AE_CE=32, /* Event cause is timer expiry */
	XFRM_AE_CU=64, /* Event cause is policy update */

How these flags are used is dependent on the direction of the
message (kernel<->user) as well the cause (config, query or event).
This is described below in the different messages.

The pid will be set appropriately in netlink to recognize direction
(0 to the kernel and pid = processid that created the event
when going from kernel to user space)

A program needs to subscribe to multicast group XFRMNLGRP_AEVENTS
to get notified of these events.

2) TLVS reflect the different parameters:
-----------------------------------------

a) byte value (XFRMA_LTIME_VAL)

This TLV carries the running/current counter for byte lifetime since
last event.

b)replay value (XFRMA_REPLAY_VAL)

This TLV carries the running/current counter for replay sequence since
last event.

c)replay threshold (XFRMA_REPLAY_THRESH)

This TLV carries the threshold being used by the kernel to trigger events
when the replay sequence is exceeded.

d) expiry timer (XFRMA_ETIMER_THRESH)

This is a timer value in milliseconds which is used as the nagle
value to rate limit the events.

3) Default configurations for the parameters:
---------------------------------------------

By default these events should be turned off unless there is
at least one listener registered to listen to the multicast
group XFRMNLGRP_AEVENTS.

Programs installing SAs will need to specify the two thresholds, however,
in order to not change existing applications such as racoon
we also provide default threshold values for these different parameters
in case they are not specified.

the two sysctls/proc entries are:

a) /proc/sys/net/core/sysctl_xfrm_aevent_etime
used to provide default values for the XFRMA_ETIMER_THRESH in incremental
units of time of 100ms. The default is 10 (1 second)

b) /proc/sys/net/core/sysctl_xfrm_aevent_rseqth
used to provide default values for XFRMA_REPLAY_THRESH parameter
in incremental packet count. The default is two packets.

4) Message types
----------------

a) XFRM_MSG_GETAE issued by user-->kernel.
   XFRM_MSG_GETAE does not carry any TLVs.

The response is a XFRM_MSG_NEWAE which is formatted based on what
XFRM_MSG_GETAE queried for.

The response will always have XFRMA_LTIME_VAL and XFRMA_REPLAY_VAL TLVs.
* if XFRM_AE_RTHR flag is set, then XFRMA_REPLAY_THRESH is also retrieved
* if XFRM_AE_ETHR flag is set, then XFRMA_ETIMER_THRESH is also retrieved

b) XFRM_MSG_NEWAE is issued by either user space to configure
   or kernel to announce events or respond to a XFRM_MSG_GETAE.

i) user --> kernel to configure a specific SA.

any of the values or threshold parameters can be updated by passing the
appropriate TLV.

A response is issued back to the sender in user space to indicate success
or failure.

In the case of success, additionally an event with
XFRM_MSG_NEWAE is also issued to any listeners as described in iii).

ii) kernel->user direction as a response to XFRM_MSG_GETAE

The response will always have XFRMA_LTIME_VAL and XFRMA_REPLAY_VAL TLVs.

The threshold TLVs will be included if explicitly requested in
the XFRM_MSG_GETAE message.

iii) kernel->user to report as event if someone sets any values or
     thresholds for an SA using XFRM_MSG_NEWAE (as described in #i above).
     In such a case XFRM_AE_CU flag is set to inform the user that
     the change happened as a result of an update.
     The message will always have XFRMA_LTIME_VAL and XFRMA_REPLAY_VAL TLVs.

iv) kernel->user to report event when replay threshold or a timeout
    is exceeded.

In such a case either XFRM_AE_CR (replay exceeded) or XFRM_AE_CE (timeout
happened) is set to inform the user what happened.
Note the two flags are mutually exclusive.
The message will always have XFRMA_LTIME_VAL and XFRMA_REPLAY_VAL TLVs.

Exceptions to threshold settings
--------------------------------

If you have an SA that is getting hit by traffic in bursts such that
there is a period where the timer threshold expires with no packets
seen, then an odd behavior is seen as follows:
The first packet arrival after a timer expiry will trigger a timeout
event; i.e we don't wait for a timeout period or a packet threshold
to be reached. This is done for simplicity and efficiency reasons.

-JHS


code:
.. SPDX-License-Identifier: GPL-2.0

============
XFRM Syscall
============

/proc/sys/net/core/xfrm_* Variables:
====================================

xfrm_acq_expires - INTEGER
	default 30 - hard timeout in seconds for acquire requests


code:
.. SPDX-License-Identifier: GPL-2.0

.. kernel-feat:: $srctree/Documentation/features nios2


code:
.. SPDX-License-Identifier: GPL-2.0

==============================
Nios II Specific Documentation
==============================

.. toctree::
   :maxdepth: 2
   :numbered:

   nios2
   features


code:
=================================
Linux on the Nios II architecture
=================================

This is a port of Linux to Nios II (nios2) processor.

In order to compile for Nios II, you need a version of GCC with support for the generic
system call ABI. Please see this link for more information on how compiling and booting
software for the Nios II platform:
http://www.rocketboards.org/foswiki/Documentation/NiosIILinuxUserManual

For reference, please see the following link:
http://www.altera.com/literature/lit-nio2.jsp

What is Nios II?
================
Nios II is a 32-bit embedded-processor architecture designed specifically for the
Altera family of FPGAs. In order to support Linux, Nios II needs to be configured
with MMU and hardware multiplier enabled.

Nios II ABI
===========
Please refer to chapter "Application Binary Interface" in Nios II Processor Reference
Handbook.


code:
LIBNVDIMM Maintainer Entry Profile
==================================

Overview
--------
The libnvdimm subsystem manages persistent memory across multiple
architectures. The mailing list is tracked by patchwork here:
https://patchwork.kernel.org/project/linux-nvdimm/list/
...and that instance is configured to give feedback to submitters on
patch acceptance and upstream merge. Patches are merged to either the
'libnvdimm-fixes' or 'libnvdimm-for-next' branch. Those branches are
available here:
https://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git/

In general patches can be submitted against the latest -rc; however, if
the incoming code change is dependent on other pending changes then the
patch should be based on the libnvdimm-for-next branch. However, since
persistent memory sits at the intersection of storage and memory there
are cases where patches are more suitable to be merged through a
Filesystem or the Memory Management tree. When in doubt copy the nvdimm
list and the maintainers will help route.

Submissions will be exposed to the kbuild robot for compile regression
testing. It helps to get a success notification from that infrastructure
before submitting, but it is not required.

Submit Checklist Addendum
-------------------------
There are unit tests for the subsystem via the ndctl utility:
https://github.com/pmem/ndctl
Those tests need to be passed before the patches go upstream, but not
necessarily before initial posting. Contact the list if you need help
getting the test environment set up.

ACPI Device Specific Methods (_DSM)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before patches enabling a new _DSM family will be considered, it must
be assigned a format-interface-code from the NVDIMM Sub-team of the ACPI
Specification Working Group. In general, the stance of the subsystem is
to push back on the proliferation of NVDIMM command sets, so do strongly
consider implementing support for an existing command set. See
drivers/acpi/nfit/nfit.h for the set of supported command sets.

Key Cycle Dates
---------------
New submissions can be sent at any time, but if they intend to hit the
next merge window they should be sent before -rc4, and ideally
stabilized in the libnvdimm-for-next branch by -rc6. Of course if a
patch set requires more than 2 weeks of review, -rc4 is already too late
and some patches may require multiple development cycles to review.

Review Cadence
--------------
In general, please wait up to one week before pinging for feedback. A
private mail reminder is preferred. Alternatively ask for other
developers that have Reviewed-by tags for libnvdimm changes to take a
look and offer their opinion.


code:
.. SPDX-License-Identifier: GPL-2.0

.. kernel-feat:: $srctree/Documentation/features openrisc


code:
.. SPDX-License-Identifier: GPL-2.0

=====================
OpenRISC Architecture
=====================

.. toctree::
   :maxdepth: 2

   openrisc_port
   todo

   features

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
==============
OpenRISC Linux
==============

This is a port of Linux to the OpenRISC class of microprocessors; the initial
target architecture, specifically, is the 32-bit OpenRISC 1000 family (or1k).

For information about OpenRISC processors and ongoing development:

	=======		=============================
	website		https://openrisc.io
	email		openrisc@lists.librecores.org
	=======		=============================

---------------------------------------------------------------------

Build instructions for OpenRISC toolchain and Linux
===================================================

In order to build and run Linux for OpenRISC, you'll need at least a basic
toolchain and, perhaps, the architectural simulator.  Steps to get these bits
in place are outlined here.

1) Toolchain

Toolchain binaries can be obtained from openrisc.io or our github releases page.
Instructions for building the different toolchains can be found on openrisc.io
or Stafford's toolchain build and release scripts.

	==========	=================================================
	binaries	https://github.com/openrisc/or1k-gcc/releases
	toolchains	https://openrisc.io/software
	building	https://github.com/stffrdhrn/or1k-toolchain-build
	==========	=================================================

2) Building

Build the Linux kernel as usual::

	make ARCH=openrisc CROSS_COMPILE="or1k-linux-" defconfig
	make ARCH=openrisc CROSS_COMPILE="or1k-linux-"

3) Running on FPGA (optional)

The OpenRISC community typically uses FuseSoC to manage building and programming
an SoC into an FPGA.  The below is an example of programming a De0 Nano
development board with the OpenRISC SoC.  During the build FPGA RTL is code
downloaded from the FuseSoC IP cores repository and built using the FPGA vendor
tools.  Binaries are loaded onto the board with openocd.

::

	git clone https://github.com/olofk/fusesoc
	cd fusesoc
	sudo pip install -e .

	fusesoc init
	fusesoc build de0_nano
	fusesoc pgm de0_nano

	openocd -f interface/altera-usb-blaster.cfg \
		-f board/or1k_generic.cfg

	telnet localhost 4444
	> init
	> halt; load_image vmlinux ; reset

4) Running on a Simulator (optional)

QEMU is a processor emulator which we recommend for simulating the OpenRISC
platform.  Please follow the OpenRISC instructions on the QEMU website to get
Linux running on QEMU.  You can build QEMU yourself, but your Linux distribution
likely provides binary packages to support OpenRISC.

	=============	======================================================
	qemu openrisc	https://wiki.qemu.org/Documentation/Platforms/OpenRISC
	=============	======================================================

---------------------------------------------------------------------

Terminology
===========

In the code, the following particles are used on symbols to limit the scope
to more or less specific processor implementations:

========= =======================================
openrisc: the OpenRISC class of processors
or1k:     the OpenRISC 1000 family of processors
or1200:   the OpenRISC 1200 processor
========= =======================================

---------------------------------------------------------------------

History
========

18-11-2003	Matjaz Breskvar (phoenix@bsemi.com)
	initial port of linux to OpenRISC/or32 architecture.
        all the core stuff is implemented and seams usable.

08-12-2003	Matjaz Breskvar (phoenix@bsemi.com)
	complete change of TLB miss handling.
	rewrite of exceptions handling.
	fully functional sash-3.6 in default initrd.
	a much improved version with changes all around.

10-04-2004	Matjaz Breskvar (phoenix@bsemi.com)
	alot of bugfixes all over.
	ethernet support, functional http and telnet servers.
	running many standard linux apps.

26-06-2004	Matjaz Breskvar (phoenix@bsemi.com)
	port to 2.6.x

30-11-2004	Matjaz Breskvar (phoenix@bsemi.com)
	lots of bugfixes and enhancments.
	added opencores framebuffer driver.

09-10-2010    Jonas Bonn (jonas@southpole.se)
	major rewrite to bring up to par with upstream Linux 2.6.36


code:
====
TODO
====

The OpenRISC Linux port is fully functional and has been tracking upstream
since 2.6.35.  There are, however, remaining items to be completed within
the coming months.  Here's a list of known-to-be-less-than-stellar items
that are due for investigation shortly, i.e. our TODO list:

-  Implement the rest of the DMA API... dma_map_sg, etc.

-  Finish the renaming cleanup... there are references to or32 in the code
   which was an older name for the architecture.  The name we've settled on is
   or1k and this change is slowly trickling through the stack.  For the time
   being, or32 is equivalent to or1k.


code:
=================
PA-RISC Debugging
=================

okay, here are some hints for debugging the lower-level parts of
linux/parisc.

1. Absolute addresses
=====================

A lot of the assembly code currently runs in real mode, which means
absolute addresses are used instead of virtual addresses as in the
rest of the kernel.  To translate an absolute address to a virtual
address you can lookup in System.map, add __PAGE_OFFSET (0x10000000
currently).

2. HPMCs
========

When real-mode code tries to access non-existent memory, you'll get
an HPMC instead of a kernel oops.  To debug an HPMC, try to find
the System Responder/Requestor addresses.  The System Requestor
address should match (one of the) processor HPAs (high addresses in
the I/O range); the System Responder address is the address real-mode
code tried to access.

Typical values for the System Responder address are addresses larger
than __PAGE_OFFSET (0x10000000) which mean a virtual address didn't
get translated to a physical address before real-mode code tried to
access it.

3. Q bit fun
============

Certain, very critical code has to clear the Q bit in the PSW.  What
happens when the Q bit is cleared is the CPU does not update the
registers interruption handlers read to find out where the machine
was interrupted - so if you get an interruption between the instruction
that clears the Q bit and the RFI that sets it again you don't know
where exactly it happened.  If you're lucky the IAOQ will point to the
instruction that cleared the Q bit, if you're not it points anywhere
at all.  Usually Q bit problems will show themselves in unexplainable
system hangs or running off the end of physical memory.


code:
.. SPDX-License-Identifier: GPL-2.0

.. kernel-feat:: $srctree/Documentation/features parisc


code:
.. SPDX-License-Identifier: GPL-2.0

====================
PA-RISC Architecture
====================

.. toctree::
   :maxdepth: 2

   debugging
   registers

   features

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
================================
Register Usage for Linux/PA-RISC
================================

[ an asterisk is used for planned usage which is currently unimplemented ]

General Registers as specified by ABI
=====================================

Control Registers
-----------------

===============================	===============================================
CR 0 (Recovery Counter)		used for ptrace
CR 1-CR 7(undefined)		unused
CR 8 (Protection ID)		per-process value*
CR 9, 12, 13 (PIDS)		unused
CR10 (CCR)			lazy FPU saving*
CR11				as specified by ABI (SAR)
CR14 (interruption vector)	initialized to fault_vector
CR15 (EIEM)			initialized to all ones*
CR16 (Interval Timer)		read for cycle count/write starts Interval Tmr
CR17-CR22			interruption parameters
CR19				Interrupt Instruction Register
CR20				Interrupt Space Register
CR21				Interrupt Offset Register
CR22				Interrupt PSW
CR23 (EIRR)			read for pending interrupts/write clears bits
CR24 (TR 0)			Kernel Space Page Directory Pointer
CR25 (TR 1)			User   Space Page Directory Pointer
CR26 (TR 2)			not used
CR27 (TR 3)			Thread descriptor pointer
CR28 (TR 4)			not used
CR29 (TR 5)			not used
CR30 (TR 6)			current / 0
CR31 (TR 7)			Temporary register, used in various places
===============================	===============================================

Space Registers (kernel mode)
-----------------------------

===============================	===============================================
SR0				temporary space register
SR4-SR7 			set to 0
SR1				temporary space register
SR2				kernel should not clobber this
SR3				used for userspace accesses (current process)
===============================	===============================================

Space Registers (user mode)
---------------------------

===============================	===============================================
SR0				temporary space register
SR1                             temporary space register
SR2                             holds space of linux gateway page
SR3                             holds user address space value while in kernel
SR4-SR7                         Defines short address space for user/kernel
===============================	===============================================

Processor Status Word
---------------------

===============================	===============================================
W (64-bit addresses)		0
E (Little-endian)		0
S (Secure Interval Timer)	0
T (Taken Branch Trap)		0
H (Higher-privilege trap)	0
L (Lower-privilege trap)	0
N (Nullify next instruction)	used by C code
X (Data memory break disable)	0
B (Taken Branch)		used by C code
C (code address translation)	1, 0 while executing real-mode code
V (divide step correction)	used by C code
M (HPMC mask)			0, 1 while executing HPMC handler*
C/B (carry/borrow bits)		used by C code
O (ordered references)		1*
F (performance monitor)		0
R (Recovery Counter trap)	0
Q (collect interruption state)	1 (0 in code directly preceding an rfi)
P (Protection Identifiers)	1*
D (Data address translation)	1, 0 while executing real-mode code
I (external interrupt mask)	used by cli()/sti() macros
===============================	===============================================

"Invisible" Registers
---------------------

===============================	===============================================
PSW default W value		0
PSW default E value		0
Shadow Registers		used by interruption handler code
TOC enable bit			1
===============================	===============================================

-------------------------------------------------------------------------

The PA-RISC architecture defines 7 registers as "shadow registers".
Those are used in RETURN FROM INTERRUPTION AND RESTORE instruction to reduce
the state save and restore time by eliminating the need for general register
(GR) saves and restores in interruption handlers.
Shadow registers are the GRs 1, 8, 9, 16, 17, 24, and 25.

-------------------------------------------------------------------------

Register usage notes, originally from John Marvin, with some additional
notes from Randolph Chung.

For the general registers:

r1,r2,r19-r26,r28,r29 & r31 can be used without saving them first. And of
course, you need to save them if you care about them, before calling
another procedure. Some of the above registers do have special meanings
that you should be aware of:

    r1:
	The addil instruction is hardwired to place its result in r1,
	so if you use that instruction be aware of that.

    r2:
	This is the return pointer. In general you don't want to
	use this, since you need the pointer to get back to your
	caller. However, it is grouped with this set of registers
	since the caller can't rely on the value being the same
	when you return, i.e. you can copy r2 to another register
	and return through that register after trashing r2, and
	that should not cause a problem for the calling routine.

    r19-r22:
	these are generally regarded as temporary registers.
	Note that in 64 bit they are arg7-arg4.

    r23-r26:
	these are arg3-arg0, i.e. you can use them if you
	don't care about the values that were passed in anymore.

    r28,r29:
	are ret0 and ret1. They are what you pass return values
	in. r28 is the primary return. When returning small structures
	r29 may also be used to pass data back to the caller.

    r30:
	stack pointer

    r31:
	the ble instruction puts the return pointer in here.

    r3-r18,r27,r30 need to be saved and restored. r3-r18 are just
    general purpose registers. r27 is the data pointer, and is
    used to make references to global variables easier. r30 is
    the stack pointer.


code:
============
Device table
============

Matching of PCMCIA devices to drivers is done using one or more of the
following criteria:

- manufactor ID
- card ID
- product ID strings _and_ hashes of these strings
- function ID
- device function (actual and pseudo)

You should use the helpers in include/pcmcia/device_id.h for generating the
struct pcmcia_device_id[] entries which match devices to drivers.

If you want to match product ID strings, you also need to pass the crc32
hashes of the string to the macro, e.g. if you want to match the product ID
string 1, you need to use

PCMCIA_DEVICE_PROD_ID1("some_string", 0x(hash_of_some_string)),

If the hash is incorrect, the kernel will inform you about this in "dmesg"
upon module initialization, and tell you of the correct hash.

You can determine the hash of the product ID strings by catting the file
"modalias" in the sysfs directory of the PCMCIA device. It generates a string
in the following form:
pcmcia:m0149cC1ABf06pfn00fn00pa725B842DpbF1EFEE84pc0877B627pd00000000

The hex value after "pa" is the hash of product ID string 1, after "pb" for
string 2 and so on.

Alternatively, you can use crc32hash (see tools/pcmcia/crc32hash.c)
to determine the crc32 hash.  Simply pass the string you want to evaluate
as argument to this program, e.g.:
$ tools/pcmcia/crc32hash "Dual Speed"


code:
==============
Driver changes
==============

This file details changes in 2.6 which affect PCMCIA card driver authors:

* pcmcia_loop_config() and autoconfiguration (as of 2.6.36)
   If `struct pcmcia_device *p_dev->config_flags` is set accordingly,
   pcmcia_loop_config() now sets up certain configuration values
   automatically, though the driver may still override the settings
   in the callback function. The following autoconfiguration options
   are provided at the moment:

	- CONF_AUTO_CHECK_VCC : check for matching Vcc
	- CONF_AUTO_SET_VPP   : set Vpp
	- CONF_AUTO_AUDIO     : auto-enable audio line, if required
	- CONF_AUTO_SET_IO    : set ioport resources (->resource[0,1])
	- CONF_AUTO_SET_IOMEM : set first iomem resource (->resource[2])

* pcmcia_request_configuration -> pcmcia_enable_device (as of 2.6.36)
   pcmcia_request_configuration() got renamed to pcmcia_enable_device(),
   as it mirrors pcmcia_disable_device(). Configuration settings are now
   stored in struct pcmcia_device, e.g. in the fields config_flags,
   config_index, config_base, vpp.

* pcmcia_request_window changes (as of 2.6.36)
   Instead of win_req_t, drivers are now requested to fill out
   `struct pcmcia_device *p_dev->resource[2,3,4,5]` for up to four ioport
   ranges. After a call to pcmcia_request_window(), the regions found there
   are reserved and may be used immediately -- until pcmcia_release_window()
   is called.

* pcmcia_request_io changes (as of 2.6.36)
   Instead of io_req_t, drivers are now requested to fill out
   `struct pcmcia_device *p_dev->resource[0,1]` for up to two ioport
   ranges. After a call to pcmcia_request_io(), the ports found there
   are reserved, after calling pcmcia_request_configuration(), they may
   be used.

* No dev_info_t, no cs_types.h (as of 2.6.36)
   dev_info_t and a few other typedefs are removed. No longer use them
   in PCMCIA device drivers. Also, do not include pcmcia/cs_types.h, as
   this file is gone.

* No dev_node_t (as of 2.6.35)
   There is no more need to fill out a "dev_node_t" structure.

* New IRQ request rules (as of 2.6.35)
   Instead of the old pcmcia_request_irq() interface, drivers may now
   choose between:

   - calling request_irq/free_irq directly. Use the IRQ from `*p_dev->irq`.
   - use pcmcia_request_irq(p_dev, handler_t); the PCMCIA core will
     clean up automatically on calls to pcmcia_disable_device() or
     device ejection.

* no cs_error / CS_CHECK / CONFIG_PCMCIA_DEBUG (as of 2.6.33)
   Instead of the cs_error() callback or the CS_CHECK() macro, please use
   Linux-style checking of return values, and -- if necessary -- debug
   messages using "dev_dbg()" or "pr_debug()".

* New CIS tuple access (as of 2.6.33)
   Instead of pcmcia_get_{first,next}_tuple(), pcmcia_get_tuple_data() and
   pcmcia_parse_tuple(), a driver shall use "pcmcia_get_tuple()" if it is
   only interested in one (raw) tuple, or "pcmcia_loop_tuple()" if it is
   interested in all tuples of one type. To decode the MAC from CISTPL_FUNCE,
   a new helper "pcmcia_get_mac_from_cis()" was added.

* New configuration loop helper (as of 2.6.28)
   By calling pcmcia_loop_config(), a driver can iterate over all available
   configuration options. During a driver's probe() phase, one doesn't need
   to use pcmcia_get_{first,next}_tuple, pcmcia_get_tuple_data and
   pcmcia_parse_tuple directly in most if not all cases.

* New release helper (as of 2.6.17)
   Instead of calling pcmcia_release_{configuration,io,irq,win}, all that's
   necessary now is calling pcmcia_disable_device. As there is no valid
   reason left to call pcmcia_release_io and pcmcia_release_irq, the
   exports for them were removed.

* Unify detach and REMOVAL event code, as well as attach and INSERTION
  code (as of 2.6.16)::

       void (*remove)          (struct pcmcia_device *dev);
       int (*probe)            (struct pcmcia_device *dev);

* Move suspend, resume and reset out of event handler (as of 2.6.16)::

       int (*suspend)          (struct pcmcia_device *dev);
       int (*resume)           (struct pcmcia_device *dev);

  should be initialized in struct pcmcia_driver, and handle
  (SUSPEND == RESET_PHYSICAL) and (RESUME == CARD_RESET) events

* event handler initialization in struct pcmcia_driver (as of 2.6.13)
   The event handler is notified of all events, and must be initialized
   as the event() callback in the driver's struct pcmcia_driver.

* pcmcia/version.h should not be used (as of 2.6.13)
   This file will be removed eventually.

* in-kernel device<->driver matching (as of 2.6.13)
   PCMCIA devices and their correct drivers can now be matched in
   kernelspace. See 'devicetable.txt' for details.

* Device model integration (as of 2.6.11)
   A struct pcmcia_device is registered with the device model core,
   and can be used (e.g. for SET_NETDEV_DEV) by using
   handle_to_dev(client_handle_t * handle).

* Convert internal I/O port addresses to unsigned int (as of 2.6.11)
   ioaddr_t should be replaced by unsigned int in PCMCIA card drivers.

* irq_mask and irq_list parameters (as of 2.6.11)
   The irq_mask and irq_list parameters should no longer be used in
   PCMCIA card drivers. Instead, it is the job of the PCMCIA core to
   determine which IRQ should be used. Therefore, link->irq.IRQInfo2
   is ignored.

* client->PendingEvents is gone (as of 2.6.11)
   client->PendingEvents is no longer available.

* client->Attributes are gone (as of 2.6.11)
   client->Attributes is unused, therefore it is removed from all
   PCMCIA card drivers

* core functions no longer available (as of 2.6.11)
   The following functions have been removed from the kernel source
   because they are unused by all in-kernel drivers, and no external
   driver was reported to rely on them::

	pcmcia_get_first_region()
	pcmcia_get_next_region()
	pcmcia_modify_window()
	pcmcia_set_event_mask()
	pcmcia_get_first_window()
	pcmcia_get_next_window()

* device list iteration upon module removal (as of 2.6.10)
   It is no longer necessary to iterate on the driver's internal
   client list and call the ->detach() function upon module removal.

* Resource management. (as of 2.6.8)
   Although the PCMCIA subsystem will allocate resources for cards,
   it no longer marks these resources busy. This means that driver
   authors are now responsible for claiming your resources as per
   other drivers in Linux. You should use request_region() to mark
   your IO regions in-use, and request_mem_region() to mark your
   memory regions in-use. The name argument should be a pointer to
   your driver name. Eg, for pcnet_cs, name should point to the
   string "pcnet_cs".

* CardServices is gone
  CardServices() in 2.4 is just a big switch statement to call various
  services.  In 2.6, all of those entry points are exported and called
  directly (except for pcmcia_report_error(), just use cs_error() instead).

* struct pcmcia_driver
  You need to use struct pcmcia_driver and pcmcia_{un,}register_driver
  instead of {un,}register_pccard_driver