Question & Answer
Question
Power9 (Model: 9009-42A) system has following configuration.
fwversion: FW860.20 (VL940_041)
AIX Level: 7200-04-02-2016
Network Adapter: ent10
PCIe3 4-Port 10GbE SR Adapter VF(df1028e21410e304)
Feature Code: EN16
entstat/netstat displays flow control counters for ent10 such as
- Number of XOFF packets transmitted
- Number of XON packets transmitted
- Number of XOFF packets received
- Number of XON packets received
- Number of Pause ON Frames Sent
- Number of Pause OFF Frames Sent
- Number of Pause ON Frames Received
- Number of Pause OFF Frames Received
How to read XON, XOFF, Pause ON, Pause OFF counters?
Answer
Ethernet flow control is a mechanism used to help manage the rate of data transfer between two devices. This is done to help prevent a sender from overwhelming a receiver by sending more packets than the receiver can handle.
The first flow control mechanism, the pause frame, is defined in the IEEE 802.3x standard. The follow-on priority-based flow control (PFC) is defined in the IEEE 802.1Qbb standard. The pause frame pauses transmission of all traffic on a physical ethernet link, while PFC selectively pauses traffic according to its class of service (CoS).
In pause frame, if device (e.g. host, switch, router etc.) receiving the data can't receive any more because receive buffers are full, then network adapter will send XOFF (i.e. Pause ON Frame) to inform the remote device to stop sending. When receive buffers have some space, network adapter sends an XON (i.e. Pause OFF Frame) to inform the remote device to start sending again.
The Priority-based Flow Control (PFC) operates on individual priorities. Instead of pausing all traffic on a link, PFC allows you to selectively pause traffic according to its CoS. The CoS is a 3-bit field called the Priority Code Point (PCP) within an Ethernet frame header when using VLAN tagged frames as defined in IEEE 802.1Q. It specifies a priority value of between 0 and 7 to differentiate traffic.
The Priority Code Point (PCP) can be configured two ways:
(1) While configuring VLAN device using "smitty vlan"
(2) While configuring SR-IOV Logical Port or vNIC Client using HMC
Flow control needs to be enabled on both devices to use it and negotiate successfully. Flow control works between two directly connected devices, and flow control frames are never forwarded between links. Thus, two hosts that are connected via a switch will never send pause frames to each other, but could send pause frames to the switch (and vice versa: the switch can send pause frames to the two hosts).
If PFC is enabled then flow control will be disabled. Any flow control frames received on a PFC configured devices are ignored.
Following snippet of entstat output explains how to read different flow control counters depending upon if SR-IOV adapter is in shared or dedicated mode.
ETHERNET STATISTICS (ent10) :
Device Type: PCIe3 4-Port 10GbE SR Adapter VF
Transmit Statistics: Receive Statistics:
--------------------- --------------------
Packets: 200356482 Packets: 862453550
Bytes: 113704269998 Bytes: 93889030245
Interrupts: 3130569 Interrupts: 639524532
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packet Dropped: 0
Bad Packets: 0
...
...
PCIe3 4-Port 10 GbE SR Adapter VF
-----------------------------------------
Device Statistics:
------------------
Device ID: df1028e21410e304
Version: 1
Physical Port Link Status: Up
Logical Port Link Status: Up
Physical Port Speed: 10 Gbps Full Duplex
...
...
Transmit and receive flow control status: Enabled
Number of XOFF packets transmitted: 0
Number of XON packets transmitted: 0
Number of XOFF packets received: 0
Number of XOFF packets received: 0
...
...
Adapter Physical Port Counters:
...
...
Number of Pause Frames Sent: 0
Number of Pause ON Frames Sent: 0
Number of Pause OFF Frames Sent: 0
...
...
Number of Pause Frames Received: 1262838
Number of Pause ON Frames Received: 1255854
Number of Pause OFF Frames Received: 6984
...
...
Adapter Logical Port Counters:
...
...
Priority Flow Control: Disabled
Traffic Class Bandwidth Priority
-------------- ------------ --------
0 100% 0 1 2 3 4 5 6 7
1 0%
2 0%
3 0%
4 0%
Traffic Class Bandwidth Priority
-------------- ------------ --------
0 100% 0 1 2 3 4 5 6 7
1 0%
2 0%
3 0%
4 0%
5 0%
6 0%
7 0%
6 0%
7 0%
• These counters are maintained by the adapter. When entstat is run, device driver fetches its physical port
counters from the adapter. The driver decides where to report those counters depending upon whether
adapter is in dedicated or shared mode.
• In dedicated mode, the device driver fills in the XON/XOFF counters section in blue and does not fill Pause
Frame counters in green.
• In shared mode, physical port is shared by multiple VFs (i.e. logical ports). When switch sends a pause
frame to physical port, it pauses transmission of all traffic on a physical ethernet link. All VFs on the
physical ports are treated the same and pause the transmission.
• In shared mode, the device driver does not fill in the XON/XOFF counters in blue because it's not possible
to know which VF caused those packets to occur. Instead, those counters are reported in the physical port
counters in green. The physical port counters are sum of all VFs if more than one VF is created from the
physical port.
• Priority-based Flow Control (PFC) is always enabled on the adapter but it will be only used when it is enabled
on the switch port and negotiated successfully. Otherwise, PFC is disabled.
• Above snippet of the entstat output shows PFC counters in red when it is negotiated:disabled.
• Below snippet of the entstat output shows PFC counters if PFC is negotiated:enabled.
Author: Darshan Patel
Operating System: AIX and VIOS
Hardware: Power
Feedback: aix_feedback@wwpdl.vnet.ibm.com
Operating System: AIX and VIOS
Hardware: Power
Feedback: aix_feedback@wwpdl.vnet.ibm.com
[{"Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z000000cw48AAA","label":"Networking->Adapters"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]
Was this topic helpful?
Document Information
Modified date:
03 November 2020
UID
ibm16347116