Agilio SmartNIC Basic Firmware User Guide¶

Support for biosdevname¶

Netronome NICs support biosdevname netdev naming with recent versions of the utility, circa December 2018, e.g. RHEL 8.0 onwards. There are some notable points to be aware of:

• Whenever an unsupported netdev is considered for naming, the biosdevname naming will be skipped and the next inline naming scheme will take preference, e.g. the systemd naming policies.
• Netdevs in breakout mode are not supported for naming.
• VF netdevs will still be subject to biosdevname naming irrespective of the breakout mode of other netdevs.
• When using an older version of the biosdevname utility, users will observe inconsistent naming of netdevs on multiport NICs, i.e. one netdev may be named according to the biosdevname scheme and another according to systemd schemes.

To disable biosdevname users can add biosdevname=0 to the kernel command line.

Refer to the online biosdevname documentation for more details about the naming policy convention that will be applied.

Upgrading firmware via the Netronome repository¶

Please refer to Appendix A: Netronome Repositories on how to configure the Netronome repository applicable to your distribution. When the repository has been successfully added install the agilio-nic-firmware package using the commands below.

Ubuntu:

# apt-get install agilio-nic-firmware
# rmmod nfp; modprobe nfp
# update-initramfs -u

CentOS/RHEL:

# yum install agilio-nic-firmware
# rmmod nfp; modprobe nfp
# dracut -f

Upgrading firmware from package installations¶

The latest firmware can be obtained at the downloads area of the Netronome Support site (https://help.netronome.com).

Install the packages provided by Netronome Support using the commands below.

Ubuntu:

# dpkg -i agilio-nic-firmware-*.deb
# rmmod nfp; modprobe nfp
# update-initramfs -u

CentOS/RHEL:

# yum install -y agilio-nic-firmware-*.rpm
# rmmod nfp; modprobe nfp
# dracut -f

Configuring interface link-speed¶

The following steps explains how to change between 10G mode and 25G mode on Agilio CX 2x25GbE cards. The changing of port speed must be done in order, p0 must be set to 10G before p1 may be set to 10G.

Down the respective interface(s):

# ip link set dev enp4s0np0 down

Set interface link-speed to 10G:

# ethtool -s enp4s0np0 speed 10000

Set interface link-speed to 25G:

# ethtool -s enp4s0np0 speed 25000

Reload driver for changes to take effect:

# rmmod nfp; modprobe nfp

Allocating IP Addresses¶

Under RHEL/Centos 7.5, the network configuration is managed by default using NetworkManager. The default configuration for unset interfaces is auto, which implies that an auto-configuration client is running on them. This means that any manual configuration made using ifconfig or iproute2 will be periodically erased.

Consult the NetworkManager documentation for detailed instructions. For example, if a connection is named ens1np0 (which corresponds to the physical port representor ens1np0 of the SmartNIC), the following commands will set the IPv4 address statically, set it to autostart on boot, and up the interface:

# nmcli c m <netdev port> ipv4.method manual
# nmcli c m <netdev port> ipv4.addresses 10.0.0.2/24
# nmcli c m <netdev port> connection.autoconnect yes
# nmcli c u <netdev port>

Alternatively, if the interface is not under control of the distribution’s network management subsystem, iproute2 can be used to configure the port:

# assign IP address to interface
# ip address add 10.0.0.2/24 dev <netdev port>
# ip link set <netdev port> up

Pinging interfaces¶

After you have successfully assigned IP addresses to the NFP interfaces perform a standard ping test to confirm connectivity:

# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.067 ms
64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=0.062 ms

Server¶

Run iPerf on the server:

# ip address add 10.0.0.1/24 dev ens1np0
# iperf -s

Client¶

Allocate an ip address on the same range as used by the server, then execute the following on the client to connect to the server and start running the test:

# iperf -c 10.0.0.1 -P 4

Example output of 1x40G link:

# iperf -c 10.0.0.1 -P 4
------------------------------------------------------------
Client connecting to 10.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  5] local 10.0.0.2 port 56938 connected with 10.0.0.1 port 5001
[  3] local 10.0.0.2 port 56932 connected with 10.0.0.1 port 5001
[  4] local 10.0.0.2 port 56934 connected with 10.0.0.1 port 5001
[  6] local 10.0.0.2 port 56936 connected with 10.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  11.9 GBytes  10.3 Gbits/sec
[  3]  0.0-10.0 sec  9.85 GBytes  8.46 Gbits/sec
[  4]  0.0-10.0 sec  11.9 GBytes  10.2 Gbits/sec
[  5]  0.0-10.0 sec  10.2 GBytes  8.75 Gbits/sec
[SUM]  0.0-10.0 sec  43.8 GBytes  37.7 Gbits/sec

View current settings¶

The -l flag can be used to view current queue/channel configuration e.g:

#ethtool -l ens1np0
Channel parameters for ens1np0:
Pre-set maximums:
RX:             20
TX:             20
Other:          2
Combined:       20
Current hardware settings:
RX:             0
TX:             12
Other:          2
Combined:       8

Configure Queues¶

The -L flag can be used to change interface queue/channel configuration. The following parameters can be configured:

rx

Receive ring interrupts

tx

Transmit ring interrupts

combined

interrupts that service both rx & tx rings

In practice use this formula to calculate parameters for the ethtool command: combined = min(RXR, TXR) ; rx = RXR - combined ; tx = TXR - combined

To configure 8 combined interrupt servicing:

# ethtool -L <intf> rx 0 tx 0 combined 8

View current hash parameters¶

The -n flag can be used to view current RSS configuration, for example by default:

# ethtool -n <netdev> rx-flow-hash tcp4
TCP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

# ethtool -n <netdev> rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA

Set hash parameters¶

The -N flag can be used to change interface RSS configuration e.g:

# ethtool -N <netdev> rx-flow-hash tcp4 sdfn
# ethtool -N <netdev> rx-flow-hash udp4 sdfn

The ethtool man pages can be consulted for full details of what RSS flags may be set

Configuring the key¶

The -x flag can be used to view current interface key configuration, for example:

# ethtool -x <intf>
# ethtool -X <intf> <hkey>

Receive Checksumming (rx-checksumming)¶

When enabled, checksum calculation and error checking comparison for received packets is offloaded to the NFP SmartNIC’s flow processor rather than the host CPU.

To enable rx-checksumming:

# ethtool -K <netdev> rx on

To disable rx-checksumming:

# ethtool -K <netdev> rx off

Transmit Checksumming (tx-checksumming)¶

When enabled, checksum calculation for outgoing packets is offloaded to the NFP SmartNIC’s flow processor rather than the host’s CPU.

To enable tx-checksumming:

# ethtool -K <netdev> tx on

To disable tx-checksumming:

# ethtool -K <netdev> tx off

Scatter and Gather (scatter-gather)¶

When enabled the NFP will use scatter and gather I/O, also known as Vectored I/O, which allows a single procedure call to sequentially read data from multiple buffers and write it to a single data stream. Only changes to the scatter-gather interface settings (from on to off or off to on) will produce a terminal output as shown below:

To enable scatter-gather:

# ethtool -K <netdev> sg on
Actual changes:
scatter-gather: on
        tx-scatter-gather: on
generic-segmentation-offload: on

To disable scatter-gather:

# ethtool -K <netdev> sg off
Actual changes:
scatter-gather: on
        tx-scatter-gather: on
generic-segmentation-offload: on

TCP Segmentation Offload (TSO)¶

When enabled, this parameter causes all functions related to the segmentation of TCP packets at egress to be offloaded to the NFP.

To enable tcp-segmentation-offload:

# ethtool -K <netdev> tso on

To disable tcp-segmentation-offload:

# ethtool -K <netdev> tso off

Generic Segmentation Offload (GSO)¶

This parameter offloads segmentation for transport layer protocol data units other than segments and datagrams for TCP/UDP respectively to the NFP. GSO operates at packet egress.

To enable generic-segmentation-offload:

# ethtool -K <netdev> gso on

To disable generic-segmentation-offload:

# ethtool -K <netdev> gso off

Generic Receive Offload (GRO)¶

This parameter enables software implementation of Large Receive Offload (LRO), which aggregates multiple packets at ingress into a large buffer before they are passed higher up the networking stack.

To enable generic-receive-offload:

# ethtool -K <netdev> gro on

To disable generic-receive-offload:

# ethtool -K <netdev> gro off

Edit grub configuration file¶

This change is required for working with VFIO, however when using kernels 4.5+, it is possible to work with VFIO and no-IOMMU mode. If your system comes with a kernel > 4.5, you can work with VFIO and no-IOMMU if desired by enabling this mode:

# echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

For kernels older than 4.5, working with VFIO requires the enabling of IOMMU in the kernel at boot time. Add the following kernel parameters to /etc/default/grub to enable IOMMU:

GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt intremap=on"

It is worth noting that iommu=pt is not required for DPDK if VFIO is used, but it does avoid a performance impact in host drivers, such as the NFP netdev driver, when intel_iommu=on is enabled.

Implement changes¶

Apply kernel parameters changes and reboot.

Ubuntu:

# update-grub2
# reboot

CentOS/RHEL:

# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

PF PMD multiport support¶

The PMD can work with up to 8 ports on the same PF device. The number of available ports is firmware and hardware dependent, and the driver looks for a firmware symbol during initialization to know how many can be used.

DPDK apps work with ports, and a port is usually a PF or a VF PCI device. However, with the NFP PF multiport there is just one PF PCI device. Supporting this particular configuration requires the PMD to create ports in a special way, although once they are created, DPDK apps should be able to use them as normal PCI ports.

NFP ports belonging to same PF can be seen inside PMD initialization with a suffix added to the PCI ID: wwww:xx:yy.z_port_n. For example, a PF with PCI ID 0000:03:00.0 and four ports is seen by the PMD code as:

0000:03:00.0_port_0
0000:03:00.0_port_1
0000:03:00.0_port_2
0000:03:00.0_port_3

Attaching vfio-pci driver¶

Load vfio-pci driver module:

# modprobe vfio-pci

Unbind current drivers:

# PCIA=0000:$(lspci -d 19ee:4000 | awk '{print $1}')
# echo $PCIA > /sys/bus/pci/devices/$PCIA/driver/unbind

Bind vfio-pci driver:

# echo 19ee 4000 > /sys/bus/pci/drivers/vfio-pci/new_id

Attaching igb-uio driver¶

Load igb-uio driver module:

# modprobe uio
# DRKO=$(find $DPDK_DIR -iname 'igb_uio.ko' | head -1 )
# insmod $DRKO

Unbind current drivers:

# PCIA=0000:$(lspci -d 19ee:4000 | awk '{print $1}')
# echo $PCIA > /sys/bus/pci/devices/$PCIA/driver/unbind

Bind igb_uio driver:

# echo 19ee 4000 > /sys/bus/pci/drivers/igb_uio/new_id

Confirm attached driver¶

Confirm that the driver has been attached:

# lspci -kd 19ee:

01:00.0 Ethernet controller: Netronome Systems, Inc. Device 4000
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel driver in use: nfp
        Kernel modules: nfp
01:08.0 Ethernet controller: Netronome Systems, Inc. Device 6003
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel driver in use: igb_uio
        Kernel modules: nfp

Unbind driver¶

Determine card address:

# PCIA=$(lspci -d 19ee: | awk '{print $1}')

Unbind vfio-pci driver:

# echo 0000:$PCIA > /sys/bus/pci/drivers/vfio-pci/unbind

Unbind igb_uio driver:

# echo 0000:$PCIA > /sys/bus/pci/drivers/igb_uio/unbind

Create default symlink¶

In order to use the PF in DPDK applications a symlink named nic_dpdk_default.nffw pointing to the applicable firmware needs to be created e.g.

Navigate to firmware directory:

# cd /lib/firmware/netronome

For Agilio 2x40G:

# cp -s nic_AMDA0097-0001_2x40.nffw nic_dpdk_default.nffw

For Agilio 2x25G:

# cp -s nic_AMDA0099-0001_2x25.nffw nic_dpdk_default.nffw

For Agilio 2x40G w/ first port in breakout mode:

# cp -s nic_AMDA0097-0001_4x10_1x40.nffw nic_dpdk_default.nffw

The following table can be used to map product names to their codes

The linux-firmware package¶

The SR-IOV capable firmware has been upstreamed into the linux-firmware package. For rpm packages, this will be available from linux-firmware 20181008-88 onwards. As of Ubuntu 18.10, the linux-firmware Debian package does not yet contain SR-IOV capable firmware.

Ensure that the latest linux-firmware package is installed.

For RHEL / Fedora / CentOS:

# yum update linux-firmware

The linux-firmware package will store the Netronome firmware files in the /lib/firmware/netronome directory. This directory contains symbolic links which point to the actual firmware files. The actual firmware files will be located in subdirectories, with each subdirectory related to a different SmartNIC functionality. Consider the following tree structure:

# tree /lib/firmware/netronome
/lib/firmware/netronome/
├── flower
│   ├── nic_AMDA0081-0001_1x40.nffw -> nic_AMDA0081.nffw
│   ├── nic_AMDA0081-0001_4x10.nffw -> nic_AMDA0081.nffw
│   ├── ...
├── nic
│   ├── nic_AMDA0058-0011_2x40.nffw
│   ├── nic_AMDA0058-0012_2x40.nffw
│   ├── ...
├── nic-sriov
│   ├── nic_AMDA0058-0011_2x40.nffw
│   ├── nic_AMDA0058-0012_2x40.nffw
│   ├── ...
├── nic_AMDA0058-0011_2x40.nffw -> nic/nic_AMDA0058-0011_2x40.nffw
├── nic_AMDA0058-0012_2x40.nffw -> nic/nic_AMDA0058-0012_2x40.nffw
├── ...

As can be seen from the tree structure, three functionalities (flower, nic, nic-sriov) are supplied by the linux-firmware package. If nic-sriov is missing, follow the The support site method. Point the symbolic links to the specific application required, in this case nic-sriov:

# ln -sf /lib/firmware/netronome/nic-sriov/* /lib/firmware/netronome/

The support site¶

The SR-IOV capable firmware can be obtained from the Netronome support site. Upon downloading the packaged firmware, install the firmware files.

For Debian / Ubuntu:

# dpkg -i agilio-sriov-firmware-2.1.x.deb

For RHEL / Fedora / CentOS:

# yum -y install agilio-sriov-firmware-2.1.x.rpm

The /lib/firmware/netronome directory contains symbolic links which point to the actual firmware files. When installing the above firmware package, the symbolic links are automatically updated to point to the new SR-IOV capable firmware files. This can be confirmed with:

# ls -og --time-style="+" /lib/firmware/netronome
...
lrwxrwxrwx 1   64  nic_AMDA0058-0011_2x40.nffw -> /opt/netronome/agilio-sriov-firmware/nic_AMDA0058-0011_2x40.nffw
...

Load firmware to SmartNIC¶

Remove and reload the driver. The driver will subsequently install the new firmware to the SmartNIC:

# modprobe -r nfp
# modprobe nfp

The ethtool command can be used to verify that the correct firmware has been loaded onto the SmartNIC:

# ethtool -i enp2s0np0 | head -3
driver: nfp
version: 4.15.0-20-generic SMP mod_unloa
firmware-version: 0.0.3.5 0.22 sriov-2.1.14 nic

Notice that the firmware has successfully changed from nic-2.0.4 to sriov-2.1.14.

Installing virtio-forwarder¶

For Debian / Ubuntu:

# add-apt-repository ppa:netronome/virtio-forwarder
# apt-get update
# apt-get install virtio-forwarder

For RHEL / Fedora / CentOS:

# yum install yum-plugin-copr
# yum copr enable netronome/virtio-forwarder
# yum install virtio-forwarder

virtio-forwarder makes use of the DPDK library, therefore DPDK has to be installed. Carry out the instructions of Installing DPDK.

Configuring hugepages¶

For Ubuntu, modify libvirt’s apparmor permissions to allow read/write access to the hugepages directory and library files for QEMU. Add the following lines to the end of /etc/apparmor.d/abstractions/libvirt-qemu:

1
2
3
4
5
6

/tmp/virtio-forwarder/** rwmix,
# for latest QEMU
/usr/lib/x86_64-linux-gnu/qemu/* rmix,
# for access to hugepages
owner "/dev/hugepages/libvirt/qemu/**" rw,
owner "/dev/hugepages-1G/libvirt/qemu/**" rw,

Also edit the existing line, such that:

1

/tmp/{,**} r,

Restart the apparmor service:

# systemctl restart apparmor.service

For virtio-forwarder, 2M hugepages are required whereas QEMU/KVM performs better with 1G hugepages. It is recommended that at least 1375 pages of 2M be reserved for virtio-forwarder. The hugepages can be configured during boot time, for which the following should be added to the Linux kernel command line parameters:

hugepagesz=2M hugepages=1375 default_hugepagesz=1G hugepagesz=1G hugepages=8

Alternatively, hugepages can be configured manually after each boot. Reserve at least 1375 * 2M for virtio-forwarder:

# echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Reserve 8G for application hugepages (modify this as needed):

# echo 8 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

Since non-fragmented memory is required for hugepages, it is recommended that hugepages be configured during boot time.

hugetlbfs needs to be mounted on the file system to allow applications to create and allocate handles to the mapped memory. The following lines mount the two types of hugepages on /dev/hugepages (2M) and /dev/hugepages-1G (1G):

# grep hugetlbfs /proc/mounts | grep -q "pagesize=2M" || \
( mkdir -p /dev/hugepages && mount nodev -t hugetlbfs -o rw,pagesize=2M /dev/hugepages/ )
# grep hugetlbfs /proc/mounts | grep -q "pagesize=1G" || \
( mkdir -p /dev/hugepages-1G && mount nodev -t hugetlbfs -o rw,pagesize=1G /dev/hugepages-1G/ )

Finally, libvirt requires a special directory inside the hugepages mounts with the correct permissions in order to create the necessary per-VM handles:

# mkdir /dev/hugepages-1G/libvirt
# mkdir /dev/hugepages/libvirt
# chown [libvirt-]qemu:kvm -R /dev/hugepages-1G/libvirt
# chown [libvirt-]qemu:kvm -R /dev/hugepages/libvirt

Restart the libvirt daemon:

# systemctl restart libvirtd

To check that hugepages are correctly reserved for each page size, the hugeadm utility can be used:

# hugeadm --pool-list

      Size  Minimum  Current  Maximum  Default
   2097152     2048     2048     2048        *
1073741824        8        8        8

Binding to vfio-pci¶

Since the VFs need to communicate directly with virtio-forwarder, a pass-through style driver, such as vfio-pci is required. The vfio-pci module is the preferred driver, compared to uio_pci_generic and igb_uio, of which the former lacks SR-IOV compatibility whereas the latter is considered outdated.

First, unbind the VF PCI devices from their current drivers:

# lspci -Dd 19ee:6003 | awk '{print $1}' | xargs -I{} echo \
"echo {} > /sys/bus/pci/devices/{}/driver/unbind;" | bash

The VFs which now have their drivers unbound, can be observed with the lspci command:

# lspci -kd 19ee:
02:00.0 Ethernet controller: Netronome Systems, Inc. Device 4000
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel driver in use: nfp
        Kernel modules: nfp
02:08.0 Ethernet controller: Netronome Systems, Inc. Device 6003
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel modules: nfp
02:08.1 Ethernet controller: Netronome Systems, Inc. Device 6003
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel modules: nfp

Notice that the Kernel driver in use attribute was removed. To bind the vfio-pci driver to the VFs, first load the vfio-pci driver to the Linux kernel:

# modprobe vfio-pci

Then bind the driver to the VFs:

# echo 19ee 6003 > /sys/bus/pci/drivers/vfio-pci/new_id

The VFs are now bound to the vfio-pci driver:

# lspci -kd 19ee:
02:00.0 Ethernet controller: Netronome Systems, Inc. Device 4000
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel driver in use: nfp
        Kernel modules: nfp
02:08.0 Ethernet controller: Netronome Systems, Inc. Device 6003
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel driver in use: vfio-pci
        Kernel modules: nfp
02:08.1 Ethernet controller: Netronome Systems, Inc. Device 6003
        Subsystem: Netronome Systems, Inc. Device 4001
        Kernel driver in use: vfio-pci
        Kernel modules: nfp

Launching virtio-forwarder¶

In this guide, the use case will be virtio-forwarder acting as a server. This means virtio-forwarder will create and host the sockets to which VMs can connect at a later stage. To configure virtio-forwarder as the server, edit /etc/default/virtioforwarder so that VIRTIOFWD_VHOST_CLIENT is assigned a blank value:

# Non-blank enables vhostuser client mode (default: server mode)
VIRTIOFWD_VHOST_CLIENT=

The virtio-forwarder service can be configured to start during boot time:

# systemctl enable virtio-forwarder

To manually start the service after installation, run:

# systemctl start virtio-forwarder

Adding VF ports to virtio-forwarder¶

Modify socket permissions:

# chown -R libvirt-qemu:kvm /tmp/virtio-forwarder/

Dynamically map the PCI address of each VF to virtio-forwarder as follows:

# /usr/lib/virtio-forwarder/virtioforwarder_port_control.py add \
--virtio-id 1 --pci-addr 02:08.0
status: OK
# /usr/lib/virtio-forwarder/virtioforwarder_port_control.py add \
--virtio-id 2 --pci-addr 02:08.1
status: OK

The virtio-id parameter is compulsory and denotes the id of the relay through which traffic is routed. A relay can accept only a single PCI device and a single VM.

The VF ports added to virtio-forwarder can be confirmed with:

# /usr/lib/virtio-forwarder/virtioforwarder_stats.py \
--include-inactive | grep DPDK_ADDED
relay_1.vf_to_vm.internal_state=DPDK_ADDED
relay_2.vf_to_vm.internal_state=DPDK_ADDED

The VF ports can be removed in a similar fashion:

# /usr/lib/virtio-forwarder/virtioforwarder_port_control.py remove \
--virtio-id 1 --pci-addr 02:08.0
status: OK
# /usr/lib/virtio-forwarder/virtioforwarder_port_control.py remove \
--virtio-id 2 --pci-addr 02:08.1
status: OK

It is useful to watch the virtio-forwarder journal while adding or removing ports:

# journalctl -fu virtio-forwarder

The VF entries can also be modified statically within the /etc/default/virtioforwarder file. Consult the virtio-forwarder docs for more information.

Modify guest VM XML files¶

The snippets in this section should be inserted in each VM’s XML file.

The following snippet configures the connection between the VM and the virtio-forwarder service. Note that virtio-forwarder1.sock refers to virtio-id 1 and relay_1. The MAC address should be assigned the value of the specific VF to be paired with the VM. If left unassigned, libvirt will assign a random MAC address which will cause the VM’s traffic to be rejected by the SmartNIC. The PCI address is internal to the VM and can be chosen arbitrarily, but should be unique within the VM itself.

1
2
3
4
5
6
7
8
9

<devices>
<interface type='vhostuser'>
    <mac address='1e:a3:32:f8:3e:83'/>
    <source type='unix' path='/tmp/virtio-forwarder/virtio-forwarder1.sock' mode='client'/>
    <model type='virtio'/>
    <alias name='net1'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</interface>
</devices>

The VM also has to be configured to make use of the 1G hugepages that was reserved for this purpose:

1
2
3
4
5

<memoryBacking>
<hugepages>
    <page size='1048576' unit='KiB' nodeset='0'/>
</hugepages>
</memoryBacking>

Allocate CPUs and memory to the VM. It is especially important to specify memAccess='shared', as this allows the host and guest VM to share RAM. This is required by virtio-forwarder to write the packets to the VM.

1
2
3
4
5
6
7

<cpu mode='custom' match='exact'>
<model fallback='allow'>SandyBridge</model>
<feature policy='require' name='ssse3'/>
<numa>
    <cell id='0' cpus='0-1' memory='3670016' unit='KiB' memAccess='shared'/>
</numa>
</cpu>

The VMs can now be booted. Observing the host’s CPU usage (e.g. htop) will show that some of the cores will be utilized to the maximum (polling mechanism). The default number of cores dedicated for virtio-forwarder is 2, and can be adjusted in /etc/default/virtioforwarder by modifying the `` VIRTIOFWD_CPU_MASK`` value.

RHEL 7.5¶

First install the required dependencies for Red Hat, DKMS is required to install the out-of-tree drivers:

# yum install -y kernel-devel-$(uname -r) elfutils-libelf-devel gcc
# wget http://fr2.rpmfind.net/linux/fedora/linux/updates/28/Everything/\
x86_64/Packages/d/dkms-2.6.1-1.fc28.noarch.rpm
# rpm -ivh dkms-2.6.1-1.fc28.noarch.rpm

Then install the NFP driver from the netronome repository added previously in Configuring repositories:

# yum list available | grep nfp-driver
agilio-nfp-driver-dkms.noarch            2017.12.18.2245.77334f7-1.el7   netronome

# yum install -y agilio-nfp-driver-dkms --nogpgcheck

RHEL/CentOS 7.5:

# yum install -y kernel-devel

# yum list available | grep nfp-driver
agilio-nfp-driver-dkms.noarch            2017.12.18.2245.77334f7-1.el7   netronome

# yum install agilio-nfp-driver-dkms --nogpgcheck

Ubuntu 18.04 LTS:

# apt-cache search nfp-driver
agilio-nfp-driver-dkms - agilio-nfp-driver driver in DKMS format.

# apt-get install agilio-nfp-driver-dkms

Kernel Changes¶

Take note that installing the DKMS driver will only install it for the currently running kernel. When you upgrade the installed kernel it may not automatically update the the nfp module to use the version in the DKMS package. In kernel versions older than v4.16 the MODULE_VERSION parameter of the in-tree module was not set, which causes DKMS to pick the module with the highest srcversion hash (https://github.com/dell/dkms/issues/14). This is worked around by the package install step adding a --force to the DKMS install, but this will not trigger on a kernel upgrade. To work around this issue, boot into the new kernel and then re-install the agilio-nfp-driver-dkms package.

This should not be a problem when upgrading from kernels v4.16 and newer as the MODULE_VERSION has been added since this revision and the DKMS version check should work properly. It’s not possible to determine which nfp.ko file was loaded by only relying on information provided by the kernel. However, it’s possible to confirm that the binary signature of a file on disk and the module loaded in memory is the same.

To confirm that the module in memory is the same as the file on disk, compare the srcversion tag. The in-memory module’s tag is at /sys/module/nfp/srcversion. The default on-disk version can be queried with modinfo

In-memory module:

# cat /sys/module/nfp/srcversion

On-disk module:

# modinfo nfp | grep "^srcversion:"

If these tags are in sync, the filename of the module provided by a modinfo query will identify the origin of the module:

# modinfo nfp | grep "^filename:"

If these tags are not in sync, there are likely conflicting copies of the module on the system: the initramfs may be out of sync or the module dependencies may be inconsistent.

The in-tree kernel module is usually located at the following path (please note, this module may be compressed with a .xz extension):

/lib/modules/$(uname -r)/kernel/drivers/net/ethernet/netronome/nfp/nfp.ko

The DKMS module is usually located at the following path:

/lib/modules/$(uname -r)/updates/dkms/nfp.ko

To ensure that the out-of-tree driver is correctly loaded instead of the in-tree module, the following commands can be run:

# mkdir -p /etc/depmod.d
# echo "override nfp * extra" > /etc/depmod.d/netronome.conf
# depmod -a
# modprobe -r nfp; modprobe nfp
# update-initramfs -u

Clone, Build and Install¶

Finally, to clone, build and install the driver:

# git clone https://github.com/Netronome/nfp-drv-kmods.git
# cd nfp-drv-kmods
# make
# make install
# depmod -a

Obtain Software¶

The latest BSP packages can be obtained at the downloads area of the Netronome Support site (https://help.netronome.com).

Install the prerequisite dependencies¶

# apt-get install -y libjansson4

NFP BSP Package¶

Install the NFP BSP package provided by Netronome Support.

RHEL/CentOS 7.5:

# yum install -y nfp-bsp-6000-*.rpm --nogpgcheck

Ubuntu 18.04 LTS:

# dpkg -i nfp-bsp-6000-*.deb

Enable CPP access¶

The NFP has an internal Command Push/Pull (CPP) bus that allows debug access to the SmartNIC internals. CPP access allows user space tools raw access to chip internals and is required to enable the use of most BSP tools. Only the out-of-tree (oot) driver allows CPP access.

Follow the steps from Install Driver via Netronome Repository to install the oot nfp driver. After the nfp module has been built load the driver with CPP access:

# depmod -a
# rmmod nfp
# modprobe nfp nfp_dev_cpp=1 nfp_pf_netdev=0

To persist this option across reboots, a number of options are available; the distribution specific documentation will detail that process more thoroughly. Care must be taken that the settings are also applied to any initramfs images generated.

Configure Media Settings¶

Alternatively to the process described in Configuring Interface Media Mode, BSP tools can be used to configure the port speed of the SmartNIC use the following commands. Note, a reboot is still required for changes to take effect.

# nfp-media phy1=10G phy0=10G

# nfp-media phy1=25G+

nfp-media --set-aneg=phy0=[S|A|I|C|F] --set-fec=phy0=[A|F|R|N]

# nfp-media phy0=40G

# nfp-media phy0=4x10G

# nfp-media phy0=40G phy1=40G

# nfp-media phy0=4x10G

# nfp-media phy0=4x10G phy1=40G

Obtain Out of Tree NFP Driver¶

To update the flash using the BSP userspace tools, use the following steps. Refer to Appendix B: Installing the Out-of-Tree NFP Driver on installing the out of tree NFP driver and to load the driver with CPP access.

Flash the Card¶

The following commands may be executed for each card installed in the system using the PCIe ID of the particular card. First reload the NFP drivers with CPP access enabled:

# rmmod nfp
# modprobe nfp nfp_pf_netdev=0 nfp_dev_cpp=1

Then use the included netronome flashing tools to reflash the card:

# /opt/netronome/bin/nfp-flash --preserve-media-overrides \
    -w /opt/netronome/flash/flash-nic.bin -Z <PCI ID, e.g. 04:00.0>
# /opt/netronome/bin/nfp-flash -w /opt/netronome/flash/flash-one.bin \
    -Z <PCI ID, e.g. 04:00.0>
# reboot

Acquire packages¶

To download the kernels from Ubuntu’s ppa mainline:

# BASE=http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18/
# wget\
    $BASE/linux-headers-4.18.0-041800_4.18.0-041800.201808122131_all.deb \
    $BASE/linux-headers-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb \
    $BASE/linux-image-unsigned-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb \
    $BASE/linux-modules-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb

Install packages¶

To install the packages:

# dpkg -i \
    linux-headers-4.18.0-041800_4.18.0-041800.201808122131_all.deb \
    linux-headers-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb \
    linux-image-unsigned-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb \
    linux-modules-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb