code:
.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)

====================================
Marvell OcteonTx2 RVU Kernel Drivers
====================================

Copyright (c) 2020 Marvell International Ltd.

Contents
========

- `Overview`_
- `Drivers`_
- `Basic packet flow`_
- `Devlink health reporters`_

Overview
========

Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
resources from the network, crypto and other functional blocks into
PCI-compatible physical and virtual functions. Each functional block
again has multiple local functions (LFs) for provisioning to PCI devices.
RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
functions (VFs). PF0 is called the administrative / admin function (AF)
and has privileges to provision RVU functional block's LFs to each of the
PF/VF.

RVU managed networking functional blocks
 - Network pool or buffer allocator (NPA)
 - Network interface controller (NIX)
 - Network parser CAM (NPC)
 - Schedule/Synchronize/Order unit (SSO)
 - Loopback interface (LBK)

RVU managed non-networking functional blocks
 - Crypto accelerator (CPT)
 - Scheduled timers unit (TIM)
 - Schedule/Synchronize/Order unit (SSO)
   Used for both networking and non networking usecases

Resource provisioning examples
 - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
 - A PF/VF with CPT-LF resource works as a pure crypto offload device.

RVU functional blocks are highly configurable as per software requirements.

Firmware setups following stuff before kernel boots
 - Enables required number of RVU PFs based on number of physical links.
 - Number of VFs per PF are either static or configurable at compile time.
   Based on config, firmware assigns VFs to each of the PFs.
 - Also assigns MSIX vectors to each of PF and VFs.
 - These are not changed after kernel boot.

Drivers
=======

Linux kernel will have multiple drivers registering to different PF and VFs
of RVU. Wrt networking there will be 3 flavours of drivers.

Admin Function driver
---------------------

As mentioned above RVU PF0 is called the admin function (AF), this driver
supports resource provisioning and configuration of functional blocks.
Doesn't handle any I/O. It sets up few basic stuff but most of the
funcionality is achieved via configuration requests from PFs and VFs.

PF/VFs communicates with AF via a shared memory region (mailbox). Upon
receiving requests AF does resource provisioning and other HW configuration.
AF is always attached to host kernel, but PFs and their VFs may be used by host
kernel itself, or attached to VMs or to userspace applications like
DPDK etc. So AF has to handle provisioning/configuration requests sent
by any device from any domain.

AF driver also interacts with underlying firmware to
 - Manage physical ethernet links ie CGX LMACs.
 - Retrieve information like speed, duplex, autoneg etc
 - Retrieve PHY EEPROM and stats.
 - Configure FEC, PAM modes
 - etc

From pure networking side AF driver supports following functionality.
 - Map a physical link to a RVU PF to which a netdev is registered.
 - Attach NIX and NPA block LFs to RVU PF/VF which provide buffer pools, RQs, SQs
   for regular networking functionality.
 - Flow control (pause frames) enable/disable/config.
 - HW PTP timestamping related config.
 - NPC parser profile config, basically how to parse pkt and what info to extract.
 - NPC extract profile config, what to extract from the pkt to match data in MCAM entries.
 - Manage NPC MCAM entries, upon request can frame and install requested packet forwarding rules.
 - Defines receive side scaling (RSS) algorithms.
 - Defines segmentation offload algorithms (eg TSO)
 - VLAN stripping, capture and insertion config.
 - SSO and TIM blocks config which provide packet scheduling support.
 - Debugfs support, to check current resource provising, current status of
   NPA pools, NIX RQ, SQ and CQs, various stats etc which helps in debugging issues.
 - And many more.

Physical Function driver
------------------------

This RVU PF handles IO, is mapped to a physical ethernet link and this
driver registers a netdev. This supports SR-IOV. As said above this driver
communicates with AF with a mailbox. To retrieve information from physical
links this driver talks to AF and AF gets that info from firmware and responds
back ie cannot talk to firmware directly.

Supports ethtool for configuring links, RSS, queue count, queue size,
flow control, ntuple filters, dump PHY EEPROM, config FEC etc.

Virtual Function driver
-----------------------

There are two types VFs, VFs that share the physical link with their parent
SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK).

Type1:
 - These VFs and their parent PF share a physical link and used for outside communication.
 - VFs cannot communicate with AF directly, they send mbox message to PF and PF
   forwards that to AF. AF after processing, responds back to PF and PF forwards
   the reply to VF.
 - From functionality point of view there is no difference between PF and VF as same type
   HW resources are attached to both. But user would be able to configure few stuff only
   from PF as PF is treated as owner/admin of the link.

Type2:
 - RVU PF0 ie admin function creates these VFs and maps them to loopback block's channels.
 - A set of two VFs (VF0 & VF1, VF2 & VF3 .. so on) works as a pair ie pkts sent out of
   VF0 will be received by VF1 and viceversa.
 - These VFs can be used by applications or virtual machines to communicate between them
   without sending traffic outside. There is no switch present in HW, hence the support
   for loopback VFs.
 - These communicate directly with AF (PF0) via mbox.

Except for the IO channels or links used for packet reception and transmission there is
no other difference between these VF types. AF driver takes care of IO channel mapping,
hence same VF driver works for both types of devices.

Basic packet flow
=================

Ingress
-------

1. CGX LMAC receives packet.
2. Forwards the packet to the NIX block.
3. Then submitted to NPC block for parsing and then MCAM lookup to get the destination RVU device.
4. NIX LF attached to the destination RVU device allocates a buffer from RQ mapped buffer pool of NPA block LF.
5. RQ may be selected by RSS or by configuring MCAM rule with a RQ number.
6. Packet is DMA'ed and driver is notified.

Egress
------

1. Driver prepares a send descriptor and submits to SQ for transmission.
2. The SQ is already configured (by AF) to transmit on a specific link/channel.
3. The SQ descriptor ring is maintained in buffers allocated from SQ mapped pool of NPA block LF.
4. NIX block transmits the pkt on the designated channel.
5. NPC MCAM entries can be installed to divert pkt onto a different channel.

Devlink health reporters
========================

NPA Reporters
-------------
The NPA reporters are responsible for reporting and recovering the following group of errors:

1. GENERAL events

   - Error due to operation of unmapped PF.
   - Error due to disabled alloc/free for other HW blocks (NIX, SSO, TIM, DPI and AURA).

2. ERROR events

   - Fault due to NPA_AQ_INST_S read or NPA_AQ_RES_S write.
   - AQ Doorbell Error.

3. RAS events

   - RAS Error Reporting for NPA_AQ_INST_S/NPA_AQ_RES_S.

4. RVU events

   - Error due to unmapped slot.

Sample Output::

	~# devlink health
	pci/0002:01:00.0:
	  reporter hw_npa_intr
	      state healthy error 2872 recover 2872 last_dump_date 2020-12-10 last_dump_time 09:39:09 grace_period 0 auto_recover true auto_dump true
	  reporter hw_npa_gen
	      state healthy error 2872 recover 2872 last_dump_date 2020-12-11 last_dump_time 04:43:04 grace_period 0 auto_recover true auto_dump true
	  reporter hw_npa_err
	      state healthy error 2871 recover 2871 last_dump_date 2020-12-10 last_dump_time 09:39:17 grace_period 0 auto_recover true auto_dump true
	   reporter hw_npa_ras
	      state healthy error 0 recover 0 last_dump_date 2020-12-10 last_dump_time 09:32:40 grace_period 0 auto_recover true auto_dump true

Each reporter dumps the

 - Error Type
 - Error Register value
 - Reason in words

For example::

	~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_gen
	 NPA_AF_GENERAL:
	         NPA General Interrupt Reg : 1
	         NIX0: free disabled RX
	~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_intr
	 NPA_AF_RVU:
	         NPA RVU Interrupt Reg : 1
	         Unmap Slot Error
	~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_err
	 NPA_AF_ERR:
	        NPA Error Interrupt Reg : 4096
	        AQ Doorbell Error

NIX Reporters
-------------
The NIX reporters are responsible for reporting and recovering the following group of errors:

1. GENERAL events

   - Receive mirror/multicast packet drop due to insufficient buffer.
   - SMQ Flush operation.

2. ERROR events

   - Memory Fault due to WQE read/write from multicast/mirror buffer.
   - Receive multicast/mirror replication list error.
   - Receive packet on an unmapped PF.
   - Fault due to NIX_AQ_INST_S read or NIX_AQ_RES_S write.
   - AQ Doorbell Error.

3. RAS events

   - RAS Error Reporting for NIX Receive Multicast/Mirror Entry Structure.
   - RAS Error Reporting for WQE/Packet Data read from Multicast/Mirror Buffer..
   - RAS Error Reporting for NIX_AQ_INST_S/NIX_AQ_RES_S.

4. RVU events

   - Error due to unmapped slot.

Sample Output::

	~# ./devlink health
	pci/0002:01:00.0:
	  reporter hw_npa_intr
	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
	  reporter hw_npa_gen
	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
	  reporter hw_npa_err
	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
	  reporter hw_npa_ras
	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
	  reporter hw_nix_intr
	    state healthy error 1121 recover 1121 last_dump_date 2021-01-19 last_dump_time 05:42:26 grace_period 0 auto_recover true auto_dump true
	  reporter hw_nix_gen
	    state healthy error 949 recover 949 last_dump_date 2021-01-19 last_dump_time 05:42:43 grace_period 0 auto_recover true auto_dump true
	  reporter hw_nix_err
	    state healthy error 1147 recover 1147 last_dump_date 2021-01-19 last_dump_time 05:42:59 grace_period 0 auto_recover true auto_dump true
	  reporter hw_nix_ras
	    state healthy error 409 recover 409 last_dump_date 2021-01-19 last_dump_time 05:43:16 grace_period 0 auto_recover true auto_dump true

Each reporter dumps the

 - Error Type
 - Error Register value
 - Reason in words

For example::

	~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_intr
	 NIX_AF_RVU:
	        NIX RVU Interrupt Reg : 1
	        Unmap Slot Error
	~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_gen
	 NIX_AF_GENERAL:
	        NIX General Interrupt Reg : 1
	        Rx multicast pkt drop
	~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_err
	 NIX_AF_ERR:
	        NIX Error Interrupt Reg : 64
	        Rx on unmapped PF_FUNC


code:
.. SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB

=================================================
Mellanox ConnectX(R) mlx5 core VPI Network Driver
=================================================

Copyright (c) 2019, Mellanox Technologies LTD.

Contents
========

- `Enabling the driver and kconfig options`_
- `Devlink info`_
- `Devlink parameters`_
- `Bridge offload`_
- `mlx5 subfunction`_
- `mlx5 function attributes`_
- `Devlink health reporters`_
- `mlx5 tracepoints`_

Enabling the driver and kconfig options
=======================================

| mlx5 core is modular and most of the major mlx5 core driver features can be selected (compiled in/out)
| at build time via kernel Kconfig flags.
| Basic features, ethernet net device rx/tx offloads and XDP, are available with the most basic flags
| CONFIG_MLX5_CORE=y/m and CONFIG_MLX5_CORE_EN=y.
| For the list of advanced features, please see below.

**CONFIG_MLX5_CORE=(y/m/n)** (module mlx5_core.ko)

|    The driver can be enabled by choosing CONFIG_MLX5_CORE=y/m in kernel config.
|    This will provide mlx5 core driver for mlx5 ulps to interface with (mlx5e, mlx5_ib).

**CONFIG_MLX5_CORE_EN=(y/n)**

|    Choosing this option will allow basic ethernet netdevice support with all of the standard rx/tx offloads.
|    mlx5e is the mlx5 ulp driver which provides netdevice kernel interface, when chosen, mlx5e will be
|    built-in into mlx5_core.ko.

**CONFIG_MLX5_EN_ARFS=(y/n)**

|     Enables Hardware-accelerated receive flow steering (arfs) support, and ntuple filtering.
|     https://community.mellanox.com/s/article/howto-configure-arfs-on-connectx-4

**CONFIG_MLX5_EN_RXNFC=(y/n)**

|    Enables ethtool receive network flow classification, which allows user defined
|    flow rules to direct traffic into arbitrary rx queue via ethtool set/get_rxnfc API.

**CONFIG_MLX5_CORE_EN_DCB=(y/n)**:

|    Enables `Data Center Bridging (DCB) Support <https://community.mellanox.com/s/article/howto-auto-config-pfc-and-ets-on-connectx-4-via-lldp-dcbx>`_.

**CONFIG_MLX5_MPFS=(y/n)**

|    Ethernet Multi-Physical Function Switch (MPFS) support in ConnectX NIC.
|    MPFs is required for when `Multi-Host <http://www.mellanox.com/page/multihost>`_ configuration is enabled to allow passing
|    user configured unicast MAC addresses to the requesting PF.

**CONFIG_MLX5_ESWITCH=(y/n)**

|    Ethernet SRIOV E-Switch support in ConnectX NIC. E-Switch provides internal SRIOV packet steering
|    and switching for the enabled VFs and PF in two available modes:
|           1) `Legacy SRIOV mode (L2 mac vlan steering based) <https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x>`_.
|           2) `Switchdev mode (eswitch offloads) <https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_User_Manual_v4.4.pdf>`_.

**CONFIG_MLX5_CORE_IPOIB=(y/n)**

|    IPoIB offloads & acceleration support.
|    Requires CONFIG_MLX5_CORE_EN to provide an accelerated interface for the rdma
|    IPoIB ulp netdevice.

**CONFIG_MLX5_FPGA=(y/n)**

|    Build support for the Innova family of network cards by Mellanox Technologies.
|    Innova network cards are comprised of a ConnectX chip and an FPGA chip on one board.
|    If you select this option, the mlx5_core driver will include the Innova FPGA core and allow
|    building sandbox-specific client drivers.

**CONFIG_MLX5_EN_IPSEC=(y/n)**

|    Enables `IPSec XFRM cryptography-offload acceleration <http://www.mellanox.com/related-docs/prod_software/Mellanox_Innova_IPsec_Ethernet_Adapter_Card_User_Manual.pdf>`_.

**CONFIG_MLX5_EN_TLS=(y/n)**

|   TLS cryptography-offload acceleration.

**CONFIG_MLX5_INFINIBAND=(y/n/m)** (module mlx5_ib.ko)

|   Provides low-level InfiniBand/RDMA and `RoCE <https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment>`_ support.

**CONFIG_MLX5_SF=(y/n)**

|   Build support for subfunction.
|   Subfunctons are more light weight than PCI SRIOV VFs. Choosing this option
|   will enable support for creating subfunction devices.

**External options** ( Choose if the corresponding mlx5 feature is required )

- CONFIG_PTP_1588_CLOCK: When chosen, mlx5 ptp support will be enabled
- CONFIG_VXLAN: When chosen, mlx5 vxlan support will be enabled.
- CONFIG_MLXFW: When chosen, mlx5 firmware flashing support will be enabled (via devlink and ethtool).

Devlink info
============

The devlink info reports the running and stored firmware versions on device.
It also prints the device PSID which represents the HCA board type ID.

User command example::

   $ devlink dev info pci/0000:00:06.0
      pci/0000:00:06.0:
      driver mlx5_core
      versions:
         fixed:
            fw.psid MT_0000000009
         running:
            fw.version 16.26.0100
         stored:
            fw.version 16.26.0100

Devlink parameters
==================

flow_steering_mode: Device flow steering mode
---------------------------------------------
The flow steering mode parameter controls the flow steering mode of the driver.
Two modes are supported:
1. 'dmfs' - Device managed flow steering.
2. 'smfs' - Software/Driver managed flow steering.

In DMFS mode, the HW steering entities are created and managed through the
Firmware.
In SMFS mode, the HW steering entities are created and managed though by
the driver directly into hardware without firmware intervention.

SMFS mode is faster and provides better rule insertion rate compared to default DMFS mode.

User command examples:

- Set SMFS flow steering mode::

    $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime

- Read device flow steering mode::

    $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
      pci/0000:06:00.0:
      name flow_steering_mode type driver-specific
      values:
         cmode runtime value smfs

enable_roce: RoCE enablement state
----------------------------------
RoCE enablement state controls driver support for RoCE traffic.
When RoCE is disabled, there is no gid table, only raw ethernet QPs are supported and traffic on the well-known UDP RoCE port is handled as raw ethernet traffic.

To change RoCE enablement state, a user must change the driverinit cmode value and run devlink reload.

User command examples:

- Disable RoCE::

    $ devlink dev param set pci/0000:06:00.0 name enable_roce value false cmode driverinit
    $ devlink dev reload pci/0000:06:00.0

- Read RoCE enablement state::

    $ devlink dev param show pci/0000:06:00.0 name enable_roce
      pci/0000:06:00.0:
      name enable_roce type generic
      values:
         cmode driverinit value true

esw_port_metadata: Eswitch port metadata state
----------------------------------------------
When applicable, disabling eswitch metadata can increase packet rate
up to 20% depending on the use case and packet sizes.

Eswitch port metadata state controls whether to internally tag packets with
metadata. Metadata tagging must be enabled for multi-port RoCE, failover
between representors and stacked devices.
By default metadata is enabled on the supported devices in E-switch.
Metadata is applicable only for E-switch in switchdev mode and
users may disable it when NONE of the below use cases will be in use:
1. HCA is in Dual/multi-port RoCE mode.
2. VF/SF representor bonding (Usually used for Live migration)
3. Stacked devices

When metadata is disabled, the above use cases will fail to initialize if
users try to enable them.

- Show eswitch port metadata::

    $ devlink dev param show pci/0000:06:00.0 name esw_port_metadata
      pci/0000:06:00.0:
        name esw_port_metadata type driver-specific
          values:
            cmode runtime value true

- Disable eswitch port metadata::

    $ devlink dev param set pci/0000:06:00.0 name esw_port_metadata value false cmode runtime

- Change eswitch mode to switchdev mode where after choosing the metadata value::

    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

Bridge offload
==============
The mlx5 driver implements support for offloading bridge rules when in switchdev
mode. Linux bridge FDBs are automatically offloaded when mlx5 switchdev
representor is attached to bridge.

- Change device to switchdev mode::

    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

- Attach mlx5 switchdev representor 'enp8s0f0' to bridge netdev 'bridge1'::

    $ ip link set enp8s0f0 master bridge1

VLANs
-----
Following bridge VLAN functions are supported by mlx5:

- VLAN filtering (including multiple VLANs per port)::

    $ ip link set bridge1 type bridge vlan_filtering 1
    $ bridge vlan add dev enp8s0f0 vid 2-3

- VLAN push on bridge ingress::

    $ bridge vlan add dev enp8s0f0 vid 3 pvid

- VLAN pop on bridge egress::

    $ bridge vlan add dev enp8s0f0 vid 3 untagged

mlx5 subfunction
================
mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.

A subfunction has its own function capabilities and its own resources. This
means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
queues are neither shared nor stolen from the parent PCI function.

When a subfunction is RDMA capable, it has its own QP1, GID table, and RDMA
resources neither shared nor stolen from the parent PCI function.

A subfunction has a dedicated window in PCI BAR space that is not shared
with the other subfunctions or the parent PCI function. This ensures that all
devices (netdev, rdma, vdpa, etc.) of the subfunction accesses only assigned
PCI BAR space.

A subfunction supports eswitch representation through which it supports tc
offloads. The user configures eswitch to send/receive packets from/to
the subfunction port.

Subfunctions share PCI level resources such as PCI MSI-X IRQs with
other subfunctions and/or with its parent PCI function.

Example mlx5 software, system, and device view::

       _______
      | admin |
      | user  |----------
      |_______|         |
          |             |
      ____|____       __|______            _________________
     |         |     |         |          |                 |
     | devlink |     | tc tool |          |    user         |
     | tool    |     |_________|          | applications    |
     |_________|         |                |_________________|
           |             |                   |          |
           |             |                   |          |         Userspace
 +---------|-------------|-------------------|----------|--------------------+
           |             |           +----------+   +----------+   Kernel
           |             |           |  netdev  |   | rdma dev |
           |             |           +----------+   +----------+
   (devlink port add/del |              ^               ^
    port function set)   |              |               |
           |             |              +---------------|
      _____|___          |              |        _______|_______
     |         |         |              |       | mlx5 class    |
     | devlink |   +------------+       |       |   drivers     |
     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
     |_________|   +------------+       |       |_______________|
           |             |              |               ^
   (devlink ops)         |              |          (probe/remove)
  _________|________     |              |           ____|________
 | subfunction      |    |     +---------------+   | subfunction |
 | management driver|-----     | subfunction   |---|  driver     |
 | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
 |__________________|          +---------------+   |_____________|
           |                                            ^
  (sf add/del, vhca events)                             |
           |                                      (device add/del)
      _____|____                                    ____|________
     |          |                                  | subfunction |
     |  PCI NIC |--- activate/deactivate events--->| host driver |
     |__________|                                  | (mlx5_core) |
                                                   |_____________|

Subfunction is created using devlink port interface.

- Change device to switchdev mode::

    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

- Add a devlink port of subfunction flavour::

    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
      function:
        hw_addr 00:00:00:00:00:00 state inactive opstate detached

- Show a devlink port of the subfunction::

    $ devlink port show pci/0000:06:00.0/32768
    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
      function:
        hw_addr 00:00:00:00:00:00 state inactive opstate detached

- Delete a devlink port of subfunction after use::

    $ devlink port del pci/0000:06:00.0/32768

mlx5 function attributes
========================
The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
a unified way for SmartNIC and non-SmartNIC.

This is supported only when the eswitch mode is set to switchdev. Port function
configuration of the PCI VF/SF is supported through devlink eswitch port.

Port function attributes should be set before PCI VF/SF is enumerated by the
driver.

MAC address setup
-----------------
mlx5 driver support devlink port function attr mechanism to setup MAC
address. (refer to Documentation/networking/devlink/devlink-port.rst)

RoCE capability setup
---------------------
Not all mlx5 PCI devices/SFs require RoCE capability.

When RoCE capability is disabled, it saves 1 Mbytes worth of system memory per
PCI devices/SF.

mlx5 driver support devlink port function attr mechanism to setup RoCE
capability. (refer to Documentation/networking/devlink/devlink-port.rst)

migratable capability setup
---------------------------
User who wants mlx5 PCI VFs to be able to perform live migration need to
explicitly enable the VF migratable capability.

mlx5 driver support devlink port function attr mechanism to setup migratable
capability. (refer to Documentation/networking/devlink/devlink-port.rst)

SF state setup
--------------
To use the SF, the user must activate the SF using the SF function state
attribute.

- Get the state of the SF identified by its unique devlink port index::

   $ devlink port show ens2f0npf0sf88
   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
     function:
       hw_addr 00:00:00:00:88:88 state inactive opstate detached

- Activate the function and verify its state is active::

   $ devlink port function set ens2f0npf0sf88 state active

   $ devlink port show ens2f0npf0sf88
   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
     function:
       hw_addr 00:00:00:00:88:88 state active opstate detached

Upon function activation, the PF driver instance gets the event from the device
that a particular SF was activated. It's the cue to put the device on bus, probe
it and instantiate the devlink instance and class specific auxiliary devices
for it.

- Show the auxiliary device and port of the subfunction::

    $ devlink dev show
    devlink dev show auxiliary/mlx5_core.sf.4

    $ devlink port show auxiliary/mlx5_core.sf.4/1
    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false

    $ rdma link show mlx5_0/1
    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88

    $ rdma dev show
    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112

- Subfunction auxiliary device and class device hierarchy::

                 mlx5_core.sf.4
          (subfunction auxiliary device)
                       /\
                      /  \
                     /    \
                    /      \
                   /        \
      mlx5_core.eth.4     mlx5_core.rdma.4
     (sf eth aux dev)     (sf rdma aux dev)
         |                      |
         |                      |
      p0sf88                  mlx5_0
     (sf netdev)          (sf rdma device)

Additionally, the SF port also gets the event when the driver attaches to the
auxiliary device of the subfunction. This results in changing the operational
state of the function. This provides visibility to the user to decide when is it
safe to delete the SF port for graceful termination of the subfunction.

- Show the SF port operational state::

    $ devlink port show ens2f0npf0sf88
    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
      function:
        hw_addr 00:00:00:00:88:88 state active opstate attached

Devlink health reporters
========================

tx reporter
-----------
The tx reporter is responsible for reporting and recovering of the following two error scenarios:

- tx timeout
    Report on kernel tx timeout detection.
    Recover by searching lost interrupts.
- tx error completion
    Report on error tx completion.
    Recover by flushing the tx queue and reset it.

tx reporter also support on demand diagnose callback, on which it provides
real time information of its send queues status.

User commands examples:

- Diagnose send queues status::

    $ devlink health diagnose pci/0000:82:00.0 reporter tx

NOTE: This command has valid output only when interface is up, otherwise the command has empty output.

- Show number of tx errors indicated, number of recover flows ended successfully,
  is autorecover enabled and graceful period from last recover::

    $ devlink health show pci/0000:82:00.0 reporter tx

rx reporter
-----------
The rx reporter is responsible for reporting and recovering of the following two error scenarios:

- rx queues' initialization (population) timeout
    Population of rx queues' descriptors on ring initialization is done
    in napi context via triggering an irq. In case of a failure to get
    the minimum amount of descriptors, a timeout would occur, and
    descriptors could be recovered by polling the EQ (Event Queue).
- rx completions with errors (reported by HW on interrupt context)
    Report on rx completion error.
    Recover (if needed) by flushing the related queue and reset it.

rx reporter also supports on demand diagnose callback, on which it
provides real time information of its receive queues' status.

- Diagnose rx queues' status and corresponding completion queue::

    $ devlink health diagnose pci/0000:82:00.0 reporter rx

NOTE: This command has valid output only when interface is up. Otherwise, the command has empty output.

- Show number of rx errors indicated, number of recover flows ended successfully,
  is autorecover enabled, and graceful period from last recover::

    $ devlink health show pci/0000:82:00.0 reporter rx

fw reporter
-----------
The fw reporter implements `diagnose` and `dump` callbacks.
It follows symptoms of fw error such as fw syndrome by triggering
fw core dump and storing it into the dump buffer.
The fw reporter diagnose command can be triggered any time by the user to check
current fw status.

User commands examples:

- Check fw heath status::

    $ devlink health diagnose pci/0000:82:00.0 reporter fw

- Read FW core dump if already stored or trigger new one::

    $ devlink health dump show pci/0000:82:00.0 reporter fw

NOTE: This command can run only on the PF which has fw tracer ownership,
running it on other PF or any VF will return "Operation not permitted".

fw fatal reporter
-----------------
The fw fatal reporter implements `dump` and `recover` callbacks.
It follows fatal errors indications by CR-space dump and recover flow.
The CR-space dump uses vsc interface which is valid even if the FW command
interface is not functional, which is the case in most FW fatal errors.
The recover function runs recover flow which reloads the driver and triggers fw
reset if needed.
On firmware error, the health buffer is dumped into the dmesg. The log
level is derived from the error's severity (given in health buffer).

User commands examples:

- Run fw recover flow manually::

    $ devlink health recover pci/0000:82:00.0 reporter fw_fatal

- Read FW CR-space dump if already stored or trigger new one::

    $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal

NOTE: This command can run only on PF.

mlx5 tracepoints
================

mlx5 driver provides internal tracepoints for tracking and debugging using
kernel tracepoints interfaces (refer to Documentation/trace/ftrace.rst).

For the list of support mlx5 events, check `/sys/kernel/debug/tracing/events/mlx5/`.

tc and eswitch offloads tracepoints:

- mlx5e_configure_flower: trace flower filter actions and cookies offloaded to mlx5::

    $ echo mlx5:mlx5e_configure_flower >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    tc-6535  [019] ...1  2672.404466: mlx5e_configure_flower: cookie=0000000067874a55 actions= REDIRECT

- mlx5e_delete_flower: trace flower filter actions and cookies deleted from mlx5::

    $ echo mlx5:mlx5e_delete_flower >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    tc-6569  [010] .N.1  2686.379075: mlx5e_delete_flower: cookie=0000000067874a55 actions= NULL

- mlx5e_stats_flower: trace flower stats request::

    $ echo mlx5:mlx5e_stats_flower >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    tc-6546  [010] ...1  2679.704889: mlx5e_stats_flower: cookie=0000000060eb3d6a bytes=0 packets=0 lastused=4295560217

- mlx5e_tc_update_neigh_used_value: trace tunnel rule neigh update value offloaded to mlx5::

    $ echo mlx5:mlx5e_tc_update_neigh_used_value >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u48:4-8806  [009] ...1 55117.882428: mlx5e_tc_update_neigh_used_value: netdev: ens1f0 IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_used=1

- mlx5e_rep_neigh_update: trace neigh update tasks scheduled due to neigh state change events::

    $ echo mlx5:mlx5e_rep_neigh_update >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u48:7-2221  [009] ...1  1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1

Bridge offloads tracepoints:

- mlx5_esw_bridge_fdb_entry_init: trace bridge FDB entry offloaded to mlx5::

    $ echo mlx5:mlx5_esw_bridge_fdb_entry_init >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u20:9-2217    [003] ...1   318.582243: mlx5_esw_bridge_fdb_entry_init: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=0 flags=0 used=0

- mlx5_esw_bridge_fdb_entry_cleanup: trace bridge FDB entry deleted from mlx5::

    $ echo mlx5:mlx5_esw_bridge_fdb_entry_cleanup >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    ip-2581    [005] ...1   318.629871: mlx5_esw_bridge_fdb_entry_cleanup: net_device=enp8s0f0_1 addr=e4:fd:05:08:00:03 vid=0 flags=0 used=16

- mlx5_esw_bridge_fdb_entry_refresh: trace bridge FDB entry offload refreshed in
  mlx5::

    $ echo mlx5:mlx5_esw_bridge_fdb_entry_refresh >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u20:8-3849    [003] ...1       466716: mlx5_esw_bridge_fdb_entry_refresh: net_device=enp8s0f0_0 addr=e4:fd:05:08:00:02 vid=3 flags=0 used=0

- mlx5_esw_bridge_vlan_create: trace bridge VLAN object add on mlx5
  representor::

    $ echo mlx5:mlx5_esw_bridge_vlan_create >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    ip-2560    [007] ...1   318.460258: mlx5_esw_bridge_vlan_create: vid=1 flags=6

- mlx5_esw_bridge_vlan_cleanup: trace bridge VLAN object delete from mlx5
  representor::

    $ echo mlx5:mlx5_esw_bridge_vlan_cleanup >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    bridge-2582    [007] ...1   318.653496: mlx5_esw_bridge_vlan_cleanup: vid=2 flags=8

- mlx5_esw_bridge_vport_init: trace mlx5 vport assigned with bridge upper
  device::

    $ echo mlx5:mlx5_esw_bridge_vport_init >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    ip-2560    [007] ...1   318.458915: mlx5_esw_bridge_vport_init: vport_num=1

- mlx5_esw_bridge_vport_cleanup: trace mlx5 vport removed from bridge upper
  device::

    $ echo mlx5:mlx5_esw_bridge_vport_cleanup >> set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    ip-5387    [000] ...1       573713: mlx5_esw_bridge_vport_cleanup: vport_num=1

Eswitch QoS tracepoints:

- mlx5_esw_vport_qos_create: trace creation of transmit scheduler arbiter for vport::

    $ echo mlx5:mlx5_esw_vport_qos_create >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    <...>-23496   [018] .... 73136.838831: mlx5_esw_vport_qos_create: (0000:82:00.0) vport=2 tsar_ix=4 bw_share=0, max_rate=0 group=000000007b576bb3

- mlx5_esw_vport_qos_config: trace configuration of transmit scheduler arbiter for vport::

    $ echo mlx5:mlx5_esw_vport_qos_config >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    <...>-26548   [023] .... 75754.223823: mlx5_esw_vport_qos_config: (0000:82:00.0) vport=1 tsar_ix=3 bw_share=34, max_rate=10000 group=000000007b576bb3

- mlx5_esw_vport_qos_destroy: trace deletion of transmit scheduler arbiter for vport::

    $ echo mlx5:mlx5_esw_vport_qos_destroy >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    <...>-27418   [004] .... 76546.680901: mlx5_esw_vport_qos_destroy: (0000:82:00.0) vport=1 tsar_ix=3

- mlx5_esw_group_qos_create: trace creation of transmit scheduler arbiter for rate group::

    $ echo mlx5:mlx5_esw_group_qos_create >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    <...>-26578   [008] .... 75776.022112: mlx5_esw_group_qos_create: (0000:82:00.0) group=000000008dac63ea tsar_ix=5

- mlx5_esw_group_qos_config: trace configuration of transmit scheduler arbiter for rate group::

    $ echo mlx5:mlx5_esw_group_qos_config >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    <...>-27303   [020] .... 76461.455356: mlx5_esw_group_qos_config: (0000:82:00.0) group=000000008dac63ea tsar_ix=5 bw_share=100 max_rate=20000

- mlx5_esw_group_qos_destroy: trace deletion of transmit scheduler arbiter for group::

    $ echo mlx5:mlx5_esw_group_qos_destroy >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    <...>-27418   [006] .... 76547.187258: mlx5_esw_group_qos_destroy: (0000:82:00.0) group=000000007b576bb3 tsar_ix=1

SF tracepoints:

- mlx5_sf_add: trace addition of the SF port::

    $ echo mlx5:mlx5_sf_add >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    devlink-9363    [031] ..... 24610.188722: mlx5_sf_add: (0000:06:00.0) port_index=32768 controller=0 hw_id=0x8000 sfnum=88

- mlx5_sf_free: trace freeing of the SF port::

    $ echo mlx5:mlx5_sf_free >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    devlink-9830    [038] ..... 26300.404749: mlx5_sf_free: (0000:06:00.0) port_index=32768 controller=0 hw_id=0x8000

- mlx5_sf_hwc_alloc: trace allocating of the hardware SF context::

    $ echo mlx5:mlx5_sf_hwc_alloc >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    devlink-9775    [031] ..... 26296.385259: mlx5_sf_hwc_alloc: (0000:06:00.0) controller=0 hw_id=0x8000 sfnum=88

- mlx5_sf_hwc_free: trace freeing of the hardware SF context::

    $ echo mlx5:mlx5_sf_hwc_free >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u128:3-9093    [046] ..... 24625.365771: mlx5_sf_hwc_free: (0000:06:00.0) hw_id=0x8000

- mlx5_sf_hwc_deferred_free : trace deferred freeing of the hardware SF context::

    $ echo mlx5:mlx5_sf_hwc_deferred_free >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    devlink-9519    [046] ..... 24624.400271: mlx5_sf_hwc_deferred_free: (0000:06:00.0) hw_id=0x8000

- mlx5_sf_vhca_event: trace SF vhca event and state::

    $ echo mlx5:mlx5_sf_vhca_event >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u128:3-9093    [046] ..... 24625.365525: mlx5_sf_vhca_event: (0000:06:00.0) hw_id=0x8000 sfnum=88 vhca_state=1

- mlx5_sf_dev_add : trace SF device add event::

    $ echo mlx5:mlx5_sf_dev_add>> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u128:3-9093    [000] ..... 24616.524495: mlx5_sf_dev_add: (0000:06:00.0) sfdev=00000000fc5d96fd aux_id=4 hw_id=0x8000 sfnum=88

- mlx5_sf_dev_del : trace SF device delete event::

    $ echo mlx5:mlx5_sf_dev_del >> /sys/kernel/debug/tracing/set_event
    $ cat /sys/kernel/debug/tracing/trace
    ...
    kworker/u128:3-9093    [044] ..... 24624.400749: mlx5_sf_dev_del: (0000:06:00.0) sfdev=00000000fc5d96fd aux_id=4 hw_id=0x8000 sfnum=88

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

======================
Hyper-V network driver
======================

Compatibility
=============

This driver is compatible with Windows Server 2012 R2, 2016 and
Windows 10.

Features
========

Checksum offload
----------------
  The netvsc driver supports checksum offload as long as the
  Hyper-V host version does. Windows Server 2016 and Azure
  support checksum offload for TCP and UDP for both IPv4 and
  IPv6. Windows Server 2012 only supports checksum offload for TCP.

Receive Side Scaling
--------------------
  Hyper-V supports receive side scaling. For TCP & UDP, packets can
  be distributed among available queues based on IP address and port
  number.

  For TCP & UDP, we can switch hash level between L3 and L4 by ethtool
  command. TCP/UDP over IPv4 and v6 can be set differently. The default
  hash level is L4. We currently only allow switching TX hash level
  from within the guests.

  On Azure, fragmented UDP packets have high loss rate with L4
  hashing. Using L3 hashing is recommended in this case.

  For example, for UDP over IPv4 on eth0:

  To include UDP port numbers in hashing::

	ethtool -N eth0 rx-flow-hash udp4 sdfn

  To exclude UDP port numbers in hashing::

	ethtool -N eth0 rx-flow-hash udp4 sd

  To show UDP hash level::

	ethtool -n eth0 rx-flow-hash udp4

Generic Receive Offload, aka GRO
--------------------------------
  The driver supports GRO and it is enabled by default. GRO coalesces
  like packets and significantly reduces CPU usage under heavy Rx
  load.

Large Receive Offload (LRO), or Receive Side Coalescing (RSC)
-------------------------------------------------------------
  The driver supports LRO/RSC in the vSwitch feature. It reduces the per packet
  processing overhead by coalescing multiple TCP segments when possible. The
  feature is enabled by default on VMs running on Windows Server 2019 and
  later. It may be changed by ethtool command::

	ethtool -K eth0 lro on
	ethtool -K eth0 lro off

SR-IOV support
--------------
  Hyper-V supports SR-IOV as a hardware acceleration option. If SR-IOV
  is enabled in both the vSwitch and the guest configuration, then the
  Virtual Function (VF) device is passed to the guest as a PCI
  device. In this case, both a synthetic (netvsc) and VF device are
  visible in the guest OS and both NIC's have the same MAC address.

  The VF is enslaved by netvsc device.  The netvsc driver will transparently
  switch the data path to the VF when it is available and up.
  Network state (addresses, firewall, etc) should be applied only to the
  netvsc device; the slave device should not be accessed directly in
  most cases.  The exceptions are if some special queue discipline or
  flow direction is desired, these should be applied directly to the
  VF slave device.

Receive Buffer
--------------
  Packets are received into a receive area which is created when device
  is probed. The receive area is broken into MTU sized chunks and each may
  contain one or more packets. The number of receive sections may be changed
  via ethtool Rx ring parameters.

  There is a similar send buffer which is used to aggregate packets
  for sending.  The send area is broken into chunks, typically of 6144
  bytes, each of section may contain one or more packets. Small
  packets are usually transmitted via copy to the send buffer. However,
  if the buffer is temporarily exhausted, or the packet to be transmitted is
  an LSO packet, the driver will provide the host with pointers to the data
  from the SKB. This attempts to achieve a balance between the overhead of
  data copy and the impact of remapping VM memory to be accessible by the
  host.

XDP support
-----------
  XDP (eXpress Data Path) is a feature that runs eBPF bytecode at the early
  stage when packets arrive at a NIC card. The goal is to increase performance
  for packet processing, reducing the overhead of SKB allocation and other
  upper network layers.

  hv_netvsc supports XDP in native mode, and transparently sets the XDP
  program on the associated VF NIC as well.

  Setting / unsetting XDP program on synthetic NIC (netvsc) propagates to
  VF NIC automatically. Setting / unsetting XDP program on VF NIC directly
  is not recommended, also not propagated to synthetic NIC, and may be
  overwritten by setting of synthetic NIC.

  XDP program cannot run with LRO (RSC) enabled, so you need to disable LRO
  before running XDP::

	ethtool -K eth0 lro off

  XDP_REDIRECT action is not yet supported.


code:
.. SPDX-License-Identifier: GPL-2.0

=========================================================
Neterion's (Formerly S2io) Xframe I/II PCI-X 10GbE driver
=========================================================

Release notes for Neterion's (Formerly S2io) Xframe I/II PCI-X 10GbE driver.

.. Contents
  - 1.  Introduction
  - 2.  Identifying the adapter/interface
  - 3.  Features supported
  - 4.  Command line parameters
  - 5.  Performance suggestions
  - 6.  Available Downloads

1. Introduction
===============
This Linux driver supports Neterion's Xframe I PCI-X 1.0 and
Xframe II PCI-X 2.0 adapters. It supports several features
such as jumbo frames, MSI/MSI-X, checksum offloads, TSO, UFO and so on.
See below for complete list of features.

All features are supported for both IPv4 and IPv6.

2. Identifying the adapter/interface
====================================

a. Insert the adapter(s) in your system.
b. Build and load driver::

	# insmod s2io.ko

c. View log messages::

	# dmesg | tail -40

You will see messages similar to::

	eth3: Neterion Xframe I 10GbE adapter (rev 3), Version 2.0.9.1, Intr type INTA
	eth4: Neterion Xframe II 10GbE adapter (rev 2), Version 2.0.9.1, Intr type INTA
	eth4: Device is on 64 bit 133MHz PCIX(M1) bus

The above messages identify the adapter type(Xframe I/II), adapter revision,
driver version, interface name(eth3, eth4), Interrupt type(INTA, MSI, MSI-X).
In case of Xframe II, the PCI/PCI-X bus width and frequency are displayed
as well.

To associate an interface with a physical adapter use "ethtool -p <ethX>".
The corresponding adapter's LED will blink multiple times.

3. Features supported
=====================
a. Jumbo frames. Xframe I/II supports MTU up to 9600 bytes,
   modifiable using ip command.

b. Offloads. Supports checksum offload(TCP/UDP/IP) on transmit
   and receive, TSO.

c. Multi-buffer receive mode. Scattering of packet across multiple
   buffers. Currently driver supports 2-buffer mode which yields
   significant performance improvement on certain platforms(SGI Altix,
   IBM xSeries).

d. MSI/MSI-X. Can be enabled on platforms which support this feature
   (IA64, Xeon) resulting in noticeable performance improvement(up to 7%
   on certain platforms).

e. Statistics. Comprehensive MAC-level and software statistics displayed
   using "ethtool -S" option.

f. Multi-FIFO/Ring. Supports up to 8 transmit queues and receive rings,
   with multiple steering options.

4. Command line parameters
==========================

a. tx_fifo_num
	Number of transmit queues

Valid range: 1-8

Default: 1

b. rx_ring_num
	Number of receive rings

Valid range: 1-8

Default: 1

c. tx_fifo_len
	Size of each transmit queue

Valid range: Total length of all queues should not exceed 8192

Default: 4096

d. rx_ring_sz
	Size of each receive ring(in 4K blocks)

Valid range: Limited by memory on system

Default: 30

e. intr_type
	Specifies interrupt type. Possible values 0(INTA), 2(MSI-X)

Valid values: 0, 2

Default: 2

5. Performance suggestions
==========================

General:

a. Set MTU to maximum(9000 for switch setup, 9600 in back-to-back configuration)
b. Set TCP windows size to optimal value.

For instance, for MTU=1500 a value of 210K has been observed to result in
good performance::

	# sysctl -w net.ipv4.tcp_rmem="210000 210000 210000"
	# sysctl -w net.ipv4.tcp_wmem="210000 210000 210000"

For MTU=9000, TCP window size of 10 MB is recommended::

	# sysctl -w net.ipv4.tcp_rmem="10000000 10000000 10000000"
	# sysctl -w net.ipv4.tcp_wmem="10000000 10000000 10000000"

Transmit performance:

a. By default, the driver respects BIOS settings for PCI bus parameters.
   However, you may want to experiment with PCI bus parameters
   max-split-transactions(MOST) and MMRBC (use setpci command).

   A MOST value of 2 has been found optimal for Opterons and 3 for Itanium.

   It could be different for your hardware.

   Set MMRBC to 4K**.

   For example you can set

   For opteron::

	#setpci -d 17d5:* 62=1d

   For Itanium::

	#setpci -d 17d5:* 62=3d

   For detailed description of the PCI registers, please see Xframe User Guide.

b. Ensure Transmit Checksum offload is enabled. Use ethtool to set/verify this
   parameter.

c. Turn on TSO(using "ethtool -K")::

	# ethtool -K <ethX> tso on

Receive performance:

a. By default, the driver respects BIOS settings for PCI bus parameters.
   However, you may want to set PCI latency timer to 248::

	#setpci -d 17d5:* LATENCY_TIMER=f8

   For detailed description of the PCI registers, please see Xframe User Guide.

b. Use 2-buffer mode. This results in large performance boost on
   certain platforms(eg. SGI Altix, IBM xSeries).

c. Ensure Receive Checksum offload is enabled. Use "ethtool -K ethX" command to
   set/verify this option.

d. Enable NAPI feature(in kernel configuration Device Drivers ---> Network
   device support --->  Ethernet (10000 Mbit) ---> S2IO 10Gbe Xframe NIC) to
   bring down CPU utilization.

.. note::

   For AMD opteron platforms with 8131 chipset, MMRBC=1 and MOST=1 are
   recommended as safe parameters.

For more information, please review the AMD8131 errata at
http://vip.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/
26310_AMD-8131_HyperTransport_PCI-X_Tunnel_Revision_Guide_rev_3_18.pdf

6. Support
==========

For further support please contact either your 10GbE Xframe NIC vendor (IBM,
HP, SGI etc.)


code:
.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
.. include:: <isonum.txt>

===========================================
Network Flow Processor (NFP) Kernel Drivers
===========================================

:Copyright: |copy| 2019, Netronome Systems, Inc.
:Copyright: |copy| 2022, Corigine, Inc.

Contents
========

- `Overview`_
- `Acquiring Firmware`_
- `Devlink Info`_
- `Configure Device`_
- `Statistics`_

Overview
========

This driver supports Netronome and Corigine's line of Network Flow Processor
devices, including the NFP3800, NFP4000, NFP5000, and NFP6000 models, which
are also incorporated in the companies' family of Agilio SmartNICs. The SR-IOV
physical and virtual functions for these devices are supported by the driver.

Acquiring Firmware
==================

The NFP3800, NFP4000 and NFP6000 devices require application specific firmware
to function. Application firmware can be located either on the host file system
or in the device flash (if supported by management firmware).

Firmware files on the host filesystem contain card type (`AMDA-*` string), media
config etc. They should be placed in `/lib/firmware/netronome` directory to
load firmware from the host file system.

Firmware for basic NIC operation is available in the upstream
`linux-firmware.git` repository.

A more comprehensive list of firmware can be downloaded from the
`Corigine Support site <https://www.corigine.com/DPUDownload.html>`_.

Firmware in NVRAM
-----------------

Recent versions of management firmware supports loading application
firmware from flash when the host driver gets probed. The firmware loading
policy configuration may be used to configure this feature appropriately.

Devlink or ethtool can be used to update the application firmware on the device
flash by providing the appropriate `nic_AMDA*.nffw` file to the respective
command. Users need to take care to write the correct firmware image for the
card and media configuration to flash.

Available storage space in flash depends on the card being used.

Dealing with multiple projects
------------------------------

NFP hardware is fully programmable therefore there can be different
firmware images targeting different applications.

When using application firmware from host, we recommend placing
actual firmware files in application-named subdirectories in
`/lib/firmware/netronome` and linking the desired files, e.g.::

    $ tree /lib/firmware/netronome/
    /lib/firmware/netronome/
    ├── bpf
    │   ├── nic_AMDA0081-0001_1x40.nffw
    │   └── nic_AMDA0081-0001_4x10.nffw
    ├── flower
    │   ├── nic_AMDA0081-0001_1x40.nffw
    │   └── nic_AMDA0081-0001_4x10.nffw
    ├── nic
    │   ├── nic_AMDA0081-0001_1x40.nffw
    │   └── nic_AMDA0081-0001_4x10.nffw
    ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw
    └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw

    3 directories, 8 files

You may need to use hard instead of symbolic links on distributions
which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu).

After changing firmware files you may need to regenerate the initramfs
image. Initramfs contains drivers and firmware files your system may
need to boot. Refer to the documentation of your distribution to find
out how to update initramfs. Good indication of stale initramfs
is system loading wrong driver or firmware on boot, but when driver is
later reloaded manually everything works correctly.

Selecting firmware per device
-----------------------------

Most commonly all cards on the system use the same type of firmware.
If you want to load a specific firmware image for a specific card, you
can use either the PCI bus address or serial number. The driver will
print which files it's looking for when it recognizes a NFP device::

    nfp: Looking for firmware file in order of priority:
    nfp:  netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found
    nfp:  netronome/pci-0000:02:00.0.nffw: not found
    nfp:  netronome/nic_AMDA0081-0001_1x40.nffw: found, loading...

In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw*
or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this
firmware file will take precedence over `nic_AMDA*` files.

Note that `serial-*` and `pci-*` files are **not** automatically included
in initramfs, you will have to refer to documentation of appropriate tools
to find out how to include them.

Running firmware version
------------------------

The version of the loaded firmware for a particular <netdev> interface,
(e.g. enp4s0), or an interface's port <netdev port> (e.g. enp4s0np0) can
be displayed with the ethtool command::

  $ ethtool -i <netdev>

Firmware loading policy
-----------------------

Firmware loading policy is controlled via three HWinfo parameters
stored as key value pairs in the device flash:

app_fw_from_flash
    Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or
    the 'Preferred' (2) firmware. When 'Preferred' is selected, the management
    firmware makes the decision over which firmware will be loaded by comparing
    versions of the flash firmware and the host supplied firmware.
    This variable is configurable using the 'fw_load_policy'
    devlink parameter.

abi_drv_reset
    Defines if the driver should reset the firmware when
    the driver is probed, either 'Disk' (0) if firmware was found on disk,
    'Always' (1) reset or 'Never' (2) reset. Note that the device is always
    reset on driver unload if firmware was loaded when the driver was probed.
    This variable is configurable using the 'reset_dev_on_drv_probe'
    devlink parameter.

abi_drv_load_ifc
    Defines a list of PF devices allowed to load FW on the device.
    This variable is not currently user configurable.

Devlink Info
============

The devlink info command displays the running and stored firmware versions
on the device, serial number and board information.

Devlink info command example (replace PCI address)::

  $ devlink dev info pci/0000:03:00.0
    pci/0000:03:00.0:
      driver nfp
      serial_number CSAAMDA2001-1003000111
      versions:
          fixed:
            board.id AMDA2001-1003
            board.rev 01
            board.manufacture CSA
            board.model mozart
          running:
            fw.mgmt 22.10.0-rc3
            fw.cpld 0x1000003
            fw.app nic-22.09.0
            chip.init AMDA-2001-1003  1003000111
          stored:
            fw.bundle_id bspbundle_1003000111
            fw.mgmt 22.10.0-rc3
            fw.cpld 0x0
            chip.init AMDA-2001-1003  1003000111

Configure Device
================

This section explains how to use Agilio SmartNICs running basic NIC firmware.

Configure interface link-speed
------------------------------
The following steps explains how to change between 10G mode and 25G mode on
Agilio CX 2x25GbE cards. The changing of port speed must be done in order,
port 0 (p0) must be set to 10G before port 1 (p1) may be set to 10G.

Down the respective interface(s)::

  $ ip link set dev <netdev port 0> down
  $ ip link set dev <netdev port 1> down

Set interface link-speed to 10G::

  $ ethtool -s <netdev port 0> speed 10000
  $ ethtool -s <netdev port 1> speed 10000

Set interface link-speed to 25G::

  $ ethtool -s <netdev port 0> speed 25000
  $ ethtool -s <netdev port 1> speed 25000

Reload driver for changes to take effect::

  $ rmmod nfp; modprobe nfp

Configure interface Maximum Transmission Unit (MTU)
---------------------------------------------------

The MTU of interfaces can temporarily be set using the iproute2, ip link or
ifconfig tools. Note that this change will not persist. Setting this via
Network Manager, or another appropriate OS configuration tool, is
recommended as changes to the MTU using Network Manager can be made to
persist.

Set interface MTU to 9000 bytes::

  $ ip link set dev <netdev port> mtu 9000

It is the responsibility of the user or the orchestration layer to set
appropriate MTU values when handling jumbo frames or utilizing tunnels. For
example, if packets sent from a VM are to be encapsulated on the card and
egress a physical port, then the MTU of the VF should be set to lower than
that of the physical port to account for the extra bytes added by the
additional header. If a setup is expected to see fallback traffic between
the SmartNIC and the kernel then the user should also ensure that the PF MTU
is appropriately set to avoid unexpected drops on this path.

Configure Forward Error Correction (FEC) modes
----------------------------------------------

Agilio SmartNICs support FEC mode configuration, e.g. Auto, Firecode Base-R,
ReedSolomon and Off modes. Each physical port's FEC mode can be set
independently using ethtool. The supported FEC modes for an interface can
be viewed using::

  $ ethtool <netdev>

The currently configured FEC mode can be viewed using::

  $ ethtool --show-fec <netdev>

To force the FEC mode for a particular port, auto-negotiation must be disabled
(see the `Auto-negotiation`_ section). An example of how to set the FEC mode
to Reed-Solomon is::

  $ ethtool --set-fec <netdev> encoding rs

Auto-negotiation
----------------

To change auto-negotiation settings, the link must first be put down. After the
link is down, auto-negotiation can be enabled or disabled using::

  ethtool -s <netdev> autoneg <on|off>

Statistics
==========

Following device statistics are available through the ``ethtool -S`` interface:

.. flat-table:: NFP device statistics
   :header-rows: 1
   :widths: 3 1 11

   * - Name
     - ID
     - Meaning

   * - dev_rx_discards
     - 1
     - Packet can be discarded on the RX path for one of the following reasons:

        * The NIC is not in promisc mode, and the destination MAC address
          doesn't match the interfaces' MAC address.
        * The received packet is larger than the max buffer size on the host.
          I.e. it exceeds the Layer 3 MRU.
        * There is no freelist descriptor available on the host for the packet.
          It is likely that the NIC couldn't cache one in time.
        * A BPF program discarded the packet.
        * The datapath drop action was executed.
        * The MAC discarded the packet due to lack of ingress buffer space
          on the NIC.

   * - dev_rx_errors
     - 2
     - A packet can be counted (and dropped) as RX error for the following
       reasons:

       * A problem with the VEB lookup (only when SR-IOV is used).
       * A physical layer problem that causes Ethernet errors, like FCS or
         alignment errors. The cause is usually faulty cables or SFPs.

   * - dev_rx_bytes
     - 3
     - Total number of bytes received.

   * - dev_rx_uc_bytes
     - 4
     - Unicast bytes received.

   * - dev_rx_mc_bytes
     - 5
     - Multicast bytes received.

   * - dev_rx_bc_bytes
     - 6
     - Broadcast bytes received.

   * - dev_rx_pkts
     - 7
     - Total number of packets received.

   * - dev_rx_mc_pkts
     - 8
     - Multicast packets received.

   * - dev_rx_bc_pkts
     - 9
     - Broadcast packets received.

   * - dev_tx_discards
     - 10
     - A packet can be discarded in the TX direction if the MAC is
       being flow controlled and the NIC runs out of TX queue space.

   * - dev_tx_errors
     - 11
     - A packet can be counted as TX error (and dropped) for one for the
       following reasons:

       * The packet is an LSO segment, but the Layer 3 or Layer 4 offset
         could not be determined. Therefore LSO could not continue.
       * An invalid packet descriptor was received over PCIe.
       * The packet Layer 3 length exceeds the device MTU.
       * An error on the MAC/physical layer. Usually due to faulty cables or
         SFPs.
       * A CTM buffer could not be allocated.
       * The packet offset was incorrect and could not be fixed by the NIC.

   * - dev_tx_bytes
     - 12
     - Total number of bytes transmitted.

   * - dev_tx_uc_bytes
     - 13
     - Unicast bytes transmitted.

   * - dev_tx_mc_bytes
     - 14
     - Multicast bytes transmitted.

   * - dev_tx_bc_bytes
     - 15
     - Broadcast bytes transmitted.

   * - dev_tx_pkts
     - 16
     - Total number of packets transmitted.

   * - dev_tx_mc_pkts
     - 17
     - Multicast packets transmitted.

   * - dev_tx_bc_pkts
     - 18
     - Broadcast packets transmitted.

Note that statistics unknown to the driver will be displayed as
``dev_unknown_stat$ID``, where ``$ID`` refers to the second column
above.


code:
.. SPDX-License-Identifier: GPL-2.0+

========================================================
Linux Driver for the Pensando(R) Ethernet adapter family
========================================================

Pensando Linux Ethernet driver.
Copyright(c) 2019 Pensando Systems, Inc

Contents
========

- Identifying the Adapter
- Enabling the driver
- Configuring the driver
- Statistics
- Support

Identifying the Adapter
=======================

To find if one or more Pensando PCI Ethernet devices are installed on the
host, check for the PCI devices::

  $ lspci -d 1dd8:
  b5:00.0 Ethernet controller: Device 1dd8:1002
  b6:00.0 Ethernet controller: Device 1dd8:1002

If such devices are listed as above, then the ionic.ko driver should find
and configure them for use.  There should be log entries in the kernel
messages such as these::

  $ dmesg | grep ionic
  ionic 0000:b5:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link)
  ionic 0000:b5:00.0 enp181s0: renamed from eth0
  ionic 0000:b5:00.0 enp181s0: Link up - 100 Gbps
  ionic 0000:b6:00.0: 126.016 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x16 link)
  ionic 0000:b6:00.0 enp182s0: renamed from eth0
  ionic 0000:b6:00.0 enp182s0: Link up - 100 Gbps

Driver and firmware version information can be gathered with either of
ethtool or devlink tools::

  $ ethtool -i enp181s0
  driver: ionic
  version: 5.7.0
  firmware-version: 1.8.0-28
  ...

  $ devlink dev info pci/0000:b5:00.0
  pci/0000:b5:00.0:
    driver ionic
    serial_number FLM18420073
    versions:
        fixed:
          asic.id 0x0
          asic.rev 0x0
        running:
          fw 1.8.0-28

See Documentation/networking/devlink/ionic.rst for more information
on the devlink dev info data.

Enabling the driver
===================

The driver is enabled via the standard kernel configuration system,
using the make command::

  make oldconfig/menuconfig/etc.

The driver is located in the menu structure at:

  -> Device Drivers
    -> Network device support (NETDEVICES [=y])
      -> Ethernet driver support
        -> Pensando devices
          -> Pensando Ethernet IONIC Support

Configuring the Driver
======================

MTU
---

Jumbo frame support is available with a maximim size of 9194 bytes.

Interrupt coalescing
--------------------

Interrupt coalescing can be configured by changing the rx-usecs value with
the "ethtool -C" command.  The rx-usecs range is 0-190.  The tx-usecs value
reflects the rx-usecs value as they are tied together on the same interrupt.

SR-IOV
------

Minimal SR-IOV support is currently offered and can be enabled by setting
the sysfs 'sriov_numvfs' value, if supported by your particular firmware
configuration.

Statistics
==========

Basic hardware stats
--------------------

The commands ``netstat -i``, ``ip -s link show``, and ``ifconfig`` show
a limited set of statistics taken directly from firmware.  For example::

  $ ip -s link show enp181s0
  7: enp181s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
      link/ether 00:ae:cd:00:07:68 brd ff:ff:ff:ff:ff:ff
      RX: bytes  packets  errors  dropped overrun mcast
      414        5        0       0       0       0
      TX: bytes  packets  errors  dropped carrier collsns
      1384       18       0       0       0       0

ethtool -S
----------

The statistics shown from the ``ethtool -S`` command includes a combination of
driver counters and firmware counters, including port and queue specific values.
The driver values are counters computed by the driver, and the firmware values
are gathered by the firmware from the port hardware and passed through the
driver with no further interpretation.

Driver port specific::

     tx_packets: 12
     tx_bytes: 964
     rx_packets: 5
     rx_bytes: 414
     tx_tso: 0
     tx_tso_bytes: 0
     tx_csum_none: 12
     tx_csum: 0
     rx_csum_none: 0
     rx_csum_complete: 3
     rx_csum_error: 0

Driver queue specific::

     tx_0_pkts: 3
     tx_0_bytes: 294
     tx_0_clean: 3
     tx_0_dma_map_err: 0
     tx_0_linearize: 0
     tx_0_frags: 0
     tx_0_tso: 0
     tx_0_tso_bytes: 0
     tx_0_csum_none: 3
     tx_0_csum: 0
     tx_0_vlan_inserted: 0
     rx_0_pkts: 2
     rx_0_bytes: 120
     rx_0_dma_map_err: 0
     rx_0_alloc_err: 0
     rx_0_csum_none: 0
     rx_0_csum_complete: 0
     rx_0_csum_error: 0
     rx_0_dropped: 0
     rx_0_vlan_stripped: 0

Firmware port specific::

     hw_tx_dropped: 0
     hw_rx_dropped: 0
     hw_rx_over_errors: 0
     hw_rx_missed_errors: 0
     hw_tx_aborted_errors: 0
     frames_rx_ok: 15
     frames_rx_all: 15
     frames_rx_bad_fcs: 0
     frames_rx_bad_all: 0
     octets_rx_ok: 1290
     octets_rx_all: 1290
     frames_rx_unicast: 10
     frames_rx_multicast: 5
     frames_rx_broadcast: 0
     frames_rx_pause: 0
     frames_rx_bad_length: 0
     frames_rx_undersized: 0
     frames_rx_oversized: 0
     frames_rx_fragments: 0
     frames_rx_jabber: 0
     frames_rx_pripause: 0
     frames_rx_stomped_crc: 0
     frames_rx_too_long: 0
     frames_rx_vlan_good: 3
     frames_rx_dropped: 0
     frames_rx_less_than_64b: 0
     frames_rx_64b: 4
     frames_rx_65b_127b: 11
     frames_rx_128b_255b: 0
     frames_rx_256b_511b: 0
     frames_rx_512b_1023b: 0
     frames_rx_1024b_1518b: 0
     frames_rx_1519b_2047b: 0
     frames_rx_2048b_4095b: 0
     frames_rx_4096b_8191b: 0
     frames_rx_8192b_9215b: 0
     frames_rx_other: 0
     frames_tx_ok: 31
     frames_tx_all: 31
     frames_tx_bad: 0
     octets_tx_ok: 2614
     octets_tx_total: 2614
     frames_tx_unicast: 8
     frames_tx_multicast: 21
     frames_tx_broadcast: 2
     frames_tx_pause: 0
     frames_tx_pripause: 0
     frames_tx_vlan: 0
     frames_tx_less_than_64b: 0
     frames_tx_64b: 4
     frames_tx_65b_127b: 27
     frames_tx_128b_255b: 0
     frames_tx_256b_511b: 0
     frames_tx_512b_1023b: 0
     frames_tx_1024b_1518b: 0
     frames_tx_1519b_2047b: 0
     frames_tx_2048b_4095b: 0
     frames_tx_4096b_8191b: 0
     frames_tx_8192b_9215b: 0
     frames_tx_other: 0
     frames_tx_pri_0: 0
     frames_tx_pri_1: 0
     frames_tx_pri_2: 0
     frames_tx_pri_3: 0
     frames_tx_pri_4: 0
     frames_tx_pri_5: 0
     frames_tx_pri_6: 0
     frames_tx_pri_7: 0
     frames_rx_pri_0: 0
     frames_rx_pri_1: 0
     frames_rx_pri_2: 0
     frames_rx_pri_3: 0
     frames_rx_pri_4: 0
     frames_rx_pri_5: 0
     frames_rx_pri_6: 0
     frames_rx_pri_7: 0
     tx_pripause_0_1us_count: 0
     tx_pripause_1_1us_count: 0
     tx_pripause_2_1us_count: 0
     tx_pripause_3_1us_count: 0
     tx_pripause_4_1us_count: 0
     tx_pripause_5_1us_count: 0
     tx_pripause_6_1us_count: 0
     tx_pripause_7_1us_count: 0
     rx_pripause_0_1us_count: 0
     rx_pripause_1_1us_count: 0
     rx_pripause_2_1us_count: 0
     rx_pripause_3_1us_count: 0
     rx_pripause_4_1us_count: 0
     rx_pripause_5_1us_count: 0
     rx_pripause_6_1us_count: 0
     rx_pripause_7_1us_count: 0
     rx_pause_1us_count: 0
     frames_tx_truncated: 0

Support
=======

For general Linux networking support, please use the netdev mailing
list, which is monitored by Pensando personnel::

  netdev@vger.kernel.org

For more specific support needs, please use the Pensando driver support
email::

  drivers@pensando.io


code:
.. SPDX-License-Identifier: GPL-2.0

================
SMC 9xxxx Driver
================

Revision 0.12

3/5/96

Copyright 1996  Erik Stahlman

Released under terms of the GNU General Public License.

This file contains the instructions and caveats for my SMC9xxx driver.  You
should not be using the driver without reading this file.

Things to note about installation:

  1. The driver should work on all kernels from 1.2.13 until 1.3.71.
     (A kernel patch is supplied for 1.3.71 )

  2. If you include this into the kernel, you might need to change some
     options, such as for forcing IRQ.

  3.  To compile as a module, run 'make'.
      Make will give you the appropriate options for various kernel support.

  4.  Loading the driver as a module::

	use:   insmod smc9194.o
	optional parameters:
		io=xxxx    : your base address
		irq=xx	   : your irq
		ifport=x   :	0 for whatever is default
				1 for twisted pair
				2 for AUI  ( or BNC on some cards )

How to obtain the latest version?

FTP:
	ftp://fenris.campus.vt.edu/smc9/smc9-12.tar.gz
	ftp://sfbox.vt.edu/filebox/F/fenris/smc9/smc9-12.tar.gz

Contacting me:
    erik@mail.vt.edu


code:
.. SPDX-License-Identifier: GPL-2.0+

==============================================================
Linux Driver for the Synopsys(R) Ethernet Controllers "stmmac"
==============================================================

Authors: Giuseppe Cavallaro <peppe.cavallaro@st.com>,
Alexandre Torgue <alexandre.torgue@st.com>, Jose Abreu <joabreu@synopsys.com>

Contents
========

- In This Release
- Feature List
- Kernel Configuration
- Command Line Parameters
- Driver Information and Notes
- Debug Information
- Support

In This Release
===============

This file describes the stmmac Linux Driver for all the Synopsys(R) Ethernet
Controllers.

Currently, this network device driver is for all STi embedded MAC/GMAC
(i.e. 7xxx/5xxx SoCs), SPEAr (arm), Loongson1B (mips) and XILINX XC2V3000
FF1152AMT0221 D1215994A VIRTEX FPGA board. The Synopsys Ethernet QoS 5.0 IPK
is also supported.

DesignWare(R) Cores Ethernet MAC 10/100/1000 Universal version 3.70a
(and older) and DesignWare(R) Cores Ethernet Quality-of-Service version 4.0
(and upper) have been used for developing this driver as well as
DesignWare(R) Cores XGMAC - 10G Ethernet MAC and DesignWare(R) Cores
Enterprise MAC - 100G Ethernet MAC.

This driver supports both the platform bus and PCI.

This driver includes support for the following Synopsys(R) DesignWare(R)
Cores Ethernet Controllers and corresponding minimum and maximum versions:

+-------------------------------+--------------+--------------+--------------+
| Controller Name               | Min. Version | Max. Version | Abbrev. Name |
+===============================+==============+==============+==============+
| Ethernet MAC Universal        | N/A          | 3.73a        | GMAC         |
+-------------------------------+--------------+--------------+--------------+
| Ethernet Quality-of-Service   | 4.00a        | N/A          | GMAC4+       |
+-------------------------------+--------------+--------------+--------------+
| XGMAC - 10G Ethernet MAC      | 2.10a        | N/A          | XGMAC2+      |
+-------------------------------+--------------+--------------+--------------+
| XLGMAC - 100G Ethernet MAC    | 2.00a        | N/A          | XLGMAC2+     |
+-------------------------------+--------------+--------------+--------------+

For questions related to hardware requirements, refer to the documentation
supplied with your Ethernet adapter. All hardware requirements listed apply
to use with Linux.

Feature List
============

The following features are available in this driver:
 - GMII/MII/RGMII/SGMII/RMII/XGMII/XLGMII Interface
 - Half-Duplex / Full-Duplex Operation
 - Energy Efficient Ethernet (EEE)
 - IEEE 802.3x PAUSE Packets (Flow Control)
 - RMON/MIB Counters
 - IEEE 1588 Timestamping (PTP)
 - Pulse-Per-Second Output (PPS)
 - MDIO Clause 22 / Clause 45 Interface
 - MAC Loopback
 - ARP Offloading
 - Automatic CRC / PAD Insertion and Checking
 - Checksum Offload for Received and Transmitted Packets
 - Standard or Jumbo Ethernet Packets
 - Source Address Insertion / Replacement
 - VLAN TAG Insertion / Replacement / Deletion / Filtering (HASH and PERFECT)
 - Programmable TX and RX Watchdog and Coalesce Settings
 - Destination Address Filtering (PERFECT)
 - HASH Filtering (Multicast)
 - Layer 3 / Layer 4 Filtering
 - Remote Wake-Up Detection
 - Receive Side Scaling (RSS)
 - Frame Preemption for TX and RX
 - Programmable Burst Length, Threshold, Queue Size
 - Multiple Queues (up to 8)
 - Multiple Scheduling Algorithms (TX: WRR, DWRR, WFQ, SP, CBS, EST, TBS;
   RX: WRR, SP)
 - Flexible RX Parser
 - TCP / UDP Segmentation Offload (TSO, USO)
 - Split Header (SPH)
 - Safety Features (ECC Protection, Data Parity Protection)
 - Selftests using Ethtool

Kernel Configuration
====================

The kernel configuration option is ``CONFIG_STMMAC_ETH``:
 - ``CONFIG_STMMAC_PLATFORM``: is to enable the platform driver.
 - ``CONFIG_STMMAC_PCI``: is to enable the pci driver.

Command Line Parameters
=======================

If the driver is built as a module the following optional parameters are used
by entering them on the command line with the modprobe command using this
syntax (e.g. for PCI module)::

    modprobe stmmac_pci [<option>=<VAL1>,<VAL2>,...]

Driver parameters can be also passed in command line by using::

    stmmaceth=watchdog:100,chain_mode=1

The default value for each parameter is generally the recommended setting,
unless otherwise noted.

watchdog
--------
:Valid Range: 5000-None
:Default Value: 5000

This parameter overrides the transmit timeout in milliseconds.

debug
-----
:Valid Range: 0-16 (0=none,...,16=all)
:Default Value: 0

This parameter adjusts the level of debug messages displayed in the system
logs.

phyaddr
-------
:Valid Range: 0-31
:Default Value: -1

This parameter overrides the physical address of the PHY device.

flow_ctrl
---------
:Valid Range: 0-3 (0=off,1=rx,2=tx,3=rx/tx)
:Default Value: 3

This parameter changes the default Flow Control ability.

pause
-----
:Valid Range: 0-65535
:Default Value: 65535

This parameter changes the default Flow Control Pause time.

tc
--
:Valid Range: 64-256
:Default Value: 64

This parameter changes the default HW FIFO Threshold control value.

buf_sz
------
:Valid Range: 1536-16384
:Default Value: 1536

This parameter changes the default RX DMA packet buffer size.

eee_timer
---------
:Valid Range: 0-None
:Default Value: 1000

This parameter changes the default LPI TX Expiration time in milliseconds.

chain_mode
----------
:Valid Range: 0-1 (0=off,1=on)
:Default Value: 0

This parameter changes the default mode of operation from Ring Mode to
Chain Mode.

Driver Information and Notes
============================

Transmit Process
----------------

The xmit method is invoked when the kernel needs to transmit a packet; it sets
the descriptors in the ring and informs the DMA engine that there is a packet
ready to be transmitted.

By default, the driver sets the ``NETIF_F_SG`` bit in the features field of
the ``net_device`` structure, enabling the scatter-gather feature. This is
true on chips and configurations where the checksum can be done in hardware.

Once the controller has finished transmitting the packet, timer will be
scheduled to release the transmit resources.

Receive Process
---------------

When one or more packets are received, an interrupt happens. The interrupts
are not queued, so the driver has to scan all the descriptors in the ring
during the receive process.

This is based on NAPI, so the interrupt handler signals only if there is work
to be done, and it exits. Then the poll method will be scheduled at some
future point.

The incoming packets are stored, by the DMA, in a list of pre-allocated socket
buffers in order to avoid the memcpy (zero-copy).

Interrupt Mitigation
--------------------

The driver is able to mitigate the number of its DMA interrupts using NAPI for
the reception on chips older than the 3.50. New chips have an HW RX Watchdog
used for this mitigation.

Mitigation parameters can be tuned by ethtool.

WoL
---

Wake up on Lan feature through Magic and Unicast frames are supported for the
GMAC, GMAC4/5 and XGMAC core.

DMA Descriptors
---------------

Driver handles both normal and alternate descriptors. The latter has been only
tested on DesignWare(R) Cores Ethernet MAC Universal version 3.41a and later.

stmmac supports DMA descriptor to operate both in dual buffer (RING) and
linked-list(CHAINED) mode. In RING each descriptor points to two data buffer
pointers whereas in CHAINED mode they point to only one data buffer pointer.
RING mode is the default.

In CHAINED mode each descriptor will have pointer to next descriptor in the
list, hence creating the explicit chaining in the descriptor itself, whereas
such explicit chaining is not possible in RING mode.

Extended Descriptors
--------------------

The extended descriptors give us information about the Ethernet payload when
it is carrying PTP packets or TCP/UDP/ICMP over IP. These are not available on
GMAC Synopsys(R) chips older than the 3.50. At probe time the driver will
decide if these can be actually used. This support also is mandatory for PTPv2
because the extra descriptors are used for saving the hardware timestamps and
Extended Status.

Ethtool Support
---------------

Ethtool is supported. For example, driver statistics (including RMON),
internal errors can be taken using::

    ethtool -S ethX

Ethtool selftests are also supported. This allows to do some early sanity
checks to the HW using MAC and PHY loopback mechanisms::

    ethtool -t ethX

Jumbo and Segmentation Offloading
---------------------------------

Jumbo frames are supported and tested for the GMAC. The GSO has been also
added but it's performed in software. LRO is not supported.

TSO Support
-----------

TSO (TCP Segmentation Offload) feature is supported by GMAC > 4.x and XGMAC
chip family. When a packet is sent through TCP protocol, the TCP stack ensures
that the SKB provided to the low level driver (stmmac in our case) matches
with the maximum frame len (IP header + TCP header + payload <= 1500 bytes
(for MTU set to 1500)). It means that if an application using TCP want to send
a packet which will have a length (after adding headers) > 1514 the packet
will be split in several TCP packets: The data payload is split and headers
(TCP/IP ..) are added. It is done by software.

When TSO is enabled, the TCP stack doesn't care about the maximum frame length
and provide SKB packet to stmmac as it is. The GMAC IP will have to perform
the segmentation by it self to match with maximum frame length.

This feature can be enabled in device tree through ``snps,tso`` entry.

Energy Efficient Ethernet
-------------------------

Energy Efficient Ethernet (EEE) enables IEEE 802.3 MAC sublayer along with a
family of Physical layer to operate in the Low Power Idle (LPI) mode. The EEE
mode supports the IEEE 802.3 MAC operation at 100Mbps, 1000Mbps and 1Gbps.

The LPI mode allows power saving by switching off parts of the communication
device functionality when there is no data to be transmitted & received.
The system on both the side of the link can disable some functionalities and
save power during the period of low-link utilization. The MAC controls whether
the system should enter or exit the LPI mode and communicate this to PHY.

As soon as the interface is opened, the driver verifies if the EEE can be
supported. This is done by looking at both the DMA HW capability register and
the PHY devices MCD registers.

To enter in TX LPI mode the driver needs to have a software timer that enable
and disable the LPI mode when there is nothing to be transmitted.

Precision Time Protocol (PTP)
-----------------------------

The driver supports the IEEE 1588-2002, Precision Time Protocol (PTP), which
enables precise synchronization of clocks in measurement and control systems
implemented with technologies such as network communication.

In addition to the basic timestamp features mentioned in IEEE 1588-2002
Timestamps, new GMAC cores support the advanced timestamp features.
IEEE 1588-2008 can be enabled when configuring the Kernel.

SGMII/RGMII Support
-------------------

New GMAC devices provide own way to manage RGMII/SGMII. This information is
available at run-time by looking at the HW capability register. This means
that the stmmac can manage auto-negotiation and link status w/o using the
PHYLIB stuff. In fact, the HW provides a subset of extended registers to
restart the ANE, verify Full/Half duplex mode and Speed. Thanks to these
registers, it is possible to look at the Auto-negotiated Link Parter Ability.

Physical
--------

The driver is compatible with Physical Abstraction Layer to be connected with
PHY and GPHY devices.

Platform Information
--------------------

Several information can be passed through the platform and device-tree.

::

    struct plat_stmmacenet_data {

1) Bus identifier::

        int bus_id;

2) PHY Physical Address. If set to -1 the driver will pick the first PHY it
finds::

        int phy_addr;

3) PHY Device Interface::

        int interface;

4) Specific platform fields for the MDIO bus::

        struct stmmac_mdio_bus_data *mdio_bus_data;

5) Internal DMA parameters::

        struct stmmac_dma_cfg *dma_cfg;

6) Fixed CSR Clock Range selection::

        int clk_csr;

7) HW uses the GMAC core::

        int has_gmac;

8) If set the MAC will use Enhanced Descriptors::

        int enh_desc;

9) Core is able to perform TX Checksum and/or RX Checksum in HW::

        int tx_coe;
        int rx_coe;

11) Some HWs are not able to perform the csum in HW for over-sized frames due
to limited buffer sizes. Setting this flag the csum will be done in SW on
JUMBO frames::

        int bugged_jumbo;

12) Core has the embedded power module::

        int pmt;

13) Force DMA to use the Store and Forward mode or Threshold mode::

        int force_sf_dma_mode;
        int force_thresh_dma_mode;

15) Force to disable the RX Watchdog feature and switch to NAPI mode::

        int riwt_off;

16) Limit the maximum operating speed and MTU::

        int max_speed;
        int maxmtu;

18) Number of Multicast/Unicast filters::

        int multicast_filter_bins;
        int unicast_filter_entries;

20) Limit the maximum TX and RX FIFO size::

        int tx_fifo_size;
        int rx_fifo_size;

21) Use the specified number of TX and RX Queues::

        u32 rx_queues_to_use;
        u32 tx_queues_to_use;

22) Use the specified TX and RX scheduling algorithm::

        u8 rx_sched_algorithm;
        u8 tx_sched_algorithm;

23) Internal TX and RX Queue parameters::

        struct stmmac_rxq_cfg rx_queues_cfg[MTL_MAX_RX_QUEUES];
        struct stmmac_txq_cfg tx_queues_cfg[MTL_MAX_TX_QUEUES];

24) This callback is used for modifying some syscfg registers (on ST SoCs)
according to the link speed negotiated by the physical layer::

        void (*fix_mac_speed)(void *priv, unsigned int speed);

25) Callbacks used for calling a custom initialization; This is sometimes
necessary on some platforms (e.g. ST boxes) where the HW needs to have set
some PIO lines or system cfg registers. init/exit callbacks should not use
or modify platform data::

        int (*init)(struct platform_device *pdev, void *priv);
        void (*exit)(struct platform_device *pdev, void *priv);

26) Perform HW setup of the bus. For example, on some ST platforms this field
is used to configure the AMBA bridge to generate more efficient STBus traffic::

        struct mac_device_info *(*setup)(void *priv);
        void *bsp_priv;

27) Internal clocks and rates::

        struct clk *stmmac_clk;
        struct clk *pclk;
        struct clk *clk_ptp_ref;
        unsigned int clk_ptp_rate;
        unsigned int clk_ref_rate;
        s32 ptp_max_adj;

28) Main reset::

        struct reset_control *stmmac_rst;

29) AXI Internal Parameters::

        struct stmmac_axi *axi;

30) HW uses GMAC>4 cores::

        int has_gmac4;

31) HW is sun8i based::

        bool has_sun8i;

32) Enables TSO feature::

        bool tso_en;

33) Enables Receive Side Scaling (RSS) feature::

        int rss_en;

34) MAC Port selection::

        int mac_port_sel_speed;

35) Enables TX LPI Clock Gating::

        bool en_tx_lpi_clockgating;

36) HW uses XGMAC>2.10 cores::

        int has_xgmac;

::

    }

For MDIO bus data, we have:

::

    struct stmmac_mdio_bus_data {

1) PHY mask passed when MDIO bus is registered::

        unsigned int phy_mask;

2) List of IRQs, one per PHY::

        int *irqs;

3) If IRQs is NULL, use this for probed PHY::

        int probed_phy_irq;

4) Set to true if PHY needs reset::

        bool needs_reset;

::

    }

For DMA engine configuration, we have:

::

    struct stmmac_dma_cfg {

1) Programmable Burst Length (TX and RX)::

        int pbl;

2) If set, DMA TX / RX will use this value rather than pbl::

        int txpbl;
        int rxpbl;

3) Enable 8xPBL::

        bool pblx8;

4) Enable Fixed or Mixed burst::

        int fixed_burst;
        int mixed_burst;

5) Enable Address Aligned Beats::

        bool aal;

6) Enable Enhanced Addressing (> 32 bits)::

        bool eame;

::

    }

For DMA AXI parameters, we have:

::

    struct stmmac_axi {

1) Enable AXI LPI::

        bool axi_lpi_en;
        bool axi_xit_frm;

2) Set AXI Write / Read maximum outstanding requests::

        u32 axi_wr_osr_lmt;
        u32 axi_rd_osr_lmt;

3) Set AXI 4KB bursts::

        bool axi_kbbe;

4) Set AXI maximum burst length map::

        u32 axi_blen[AXI_BLEN];

5) Set AXI Fixed burst / mixed burst::

        bool axi_fb;
        bool axi_mb;

6) Set AXI rebuild incrx mode::

        bool axi_rb;

::

    }

For the RX Queues configuration, we have:

::

    struct stmmac_rxq_cfg {

1) Mode to use (DCB or AVB)::

        u8 mode_to_use;

2) DMA channel to use::

        u32 chan;

3) Packet routing, if applicable::

        u8 pkt_route;

4) Use priority routing, and priority to route::

        bool use_prio;
        u32 prio;

::

    }

For the TX Queues configuration, we have:

::

    struct stmmac_txq_cfg {

1) Queue weight in scheduler::

        u32 weight;

2) Mode to use (DCB or AVB)::

        u8 mode_to_use;

3) Credit Base Shaper Parameters::

        u32 send_slope;
        u32 idle_slope;
        u32 high_credit;
        u32 low_credit;

4) Use priority scheduling, and priority::

        bool use_prio;
        u32 prio;

::

    }

Device Tree Information
-----------------------

Please refer to the following document:
Documentation/devicetree/bindings/net/snps,dwmac.yaml

HW Capabilities
---------------

Note that, starting from new chips, where it is available the HW capability
register, many configurations are discovered at run-time for example to
understand if EEE, HW csum, PTP, enhanced descriptor etc are actually
available. As strategy adopted in this driver, the information from the HW
capability register can replace what has been passed from the platform.

Debug Information
=================

The driver exports many information i.e. internal statistics, debug
information, MAC and DMA registers etc.

These can be read in several ways depending on the type of the information
actually needed.

For example a user can be use the ethtool support to get statistics: e.g.
using: ``ethtool -S ethX`` (that shows the Management counters (MMC) if
supported) or sees the MAC/DMA registers: e.g. using: ``ethtool -d ethX``

Compiling the Kernel with ``CONFIG_DEBUG_FS`` the driver will export the
following debugfs entries:

 - ``descriptors_status``: To show the DMA TX/RX descriptor rings
 - ``dma_cap``: To show the HW Capabilities

Developer can also use the ``debug`` module parameter to get further debug
information (please see: NETIF Msg Level).

Support
=======

If an issue is identified with the released source code on a supported kernel
with a supported adapter, email the specific information related to the
issue to netdev@vger.kernel.org


code:
.. SPDX-License-Identifier: GPL-2.0

===================================================================
Texas Instruments K3 AM65 CPSW NUSS switchdev based ethernet driver
===================================================================

:Version: 1.0

Port renaming
=============

In order to rename via udev::

    ip -d link show dev sw0p1 | grep switchid

    SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}==<switchid>, \
	    ATTR{phys_port_name}!="", NAME="sw0$attr{phys_port_name}"

Multi mac mode
==============

- The driver is operating in multi-mac mode by default, thus
  working as N individual network interfaces.

Devlink configuration parameters
================================

See Documentation/networking/devlink/am65-nuss-cpsw-switch.rst

Enabling "switch"
=================

The Switch mode can be enabled by configuring devlink driver parameter
"switch_mode" to 1/true::

        devlink dev param set platform/c000000.ethernet \
        name switch_mode value true cmode runtime

This can be done regardless of the state of Port's netdev devices - UP/DOWN, but
Port's netdev devices have to be in UP before joining to the bridge to avoid
overwriting of bridge configuration as CPSW switch driver completely reloads its
configuration when first port changes its state to UP.

When the both interfaces joined the bridge - CPSW switch driver will enable
marking packets with offload_fwd_mark flag.

All configuration is implemented via switchdev API.

Bridge setup
============

::

        devlink dev param set platform/c000000.ethernet \
        name switch_mode value true cmode runtime

	ip link add name br0 type bridge
	ip link set dev br0 type bridge ageing_time 1000
	ip link set dev sw0p1 up
	ip link set dev sw0p2 up
	ip link set dev sw0p1 master br0
	ip link set dev sw0p2 master br0

	[*] bridge vlan add dev br0 vid 1 pvid untagged self

	[*] if vlan_filtering=1. where default_pvid=1

	Note. Steps [*] are mandatory.

On/off STP
==========

::

	ip link set dev BRDEV type bridge stp_state 1/0

VLAN configuration
==================

::

  bridge vlan add dev br0 vid 1 pvid untagged self <---- add cpu port to VLAN 1

Note. This step is mandatory for bridge/default_pvid.

Add extra VLANs
===============

 1. untagged::

	bridge vlan add dev sw0p1 vid 100 pvid untagged master
	bridge vlan add dev sw0p2 vid 100 pvid untagged master
	bridge vlan add dev br0 vid 100 pvid untagged self <---- Add cpu port to VLAN100

 2. tagged::

	bridge vlan add dev sw0p1 vid 100 master
	bridge vlan add dev sw0p2 vid 100 master
	bridge vlan add dev br0 vid 100 pvid tagged self <---- Add cpu port to VLAN100

FDBs
----

FDBs are automatically added on the appropriate switch port upon detection

Manually adding FDBs::

    bridge fdb add aa:bb:cc:dd:ee:ff dev sw0p1 master vlan 100
    bridge fdb add aa:bb:cc:dd:ee:fe dev sw0p2 master <---- Add on all VLANs

MDBs
----

MDBs are automatically added on the appropriate switch port upon detection

Manually adding MDBs::

  bridge mdb add dev br0 port sw0p1 grp 239.1.1.1 permanent vid 100
  bridge mdb add dev br0 port sw0p1 grp 239.1.1.1 permanent <---- Add on all VLANs

Multicast flooding
==================
CPU port mcast_flooding is always on

Turning flooding on/off on swithch ports:
bridge link set dev sw0p1 mcast_flood on/off

Access and Trunk port
=====================

::

 bridge vlan add dev sw0p1 vid 100 pvid untagged master
 bridge vlan add dev sw0p2 vid 100 master

 bridge vlan add dev br0 vid 100 self
 ip link add link br0 name br0.100 type vlan id 100

Note. Setting PVID on Bridge device itself works only for
default VLAN (default_pvid).