IBM Support

Linux on POWER Ethernet adapters: Which is Which?

How To


Summary

PCIe adapter slots and Ethernet port name can easily get confusing, this article might help.

Objective

Nigels Banner

Steps

So I have a Linux on Power Scale-Out S822L Server with VM with physical adapters but which Ethernet adapter is which after I added two more?

First port of call is the regular lspci command:


# lspci
0000:70:00.0 RAID bus controller: IBM PCI-E IPR SAS Adapter (ASIC) (rev 02)
0001:80:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0001:80:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0001:80:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0001:80:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0002:01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
0003:01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
#

My two new Mellanox adapters are there at the end but which is in Slot C2 and in C5 and what are they new fangled Ethernet device names (like the old eth0 etc)?

The old fashioned eth0 sort of names could be renumbered if added the adapters in to earlier PCI slot numbers and cause chaos.

The 0002:01:00.0 and 0003:01:00.0 probably mean something - the manuals say the format is:

  • domain:bus:device.function Class: Vendor  Device and optional PhySlot.

So

  • Vendor is Mellanox Technologies
  • Device is MT27520 Family [ConnectX-3 Pro]

So the Physical Slot number is missing. Darn!

I installed the lsvpd command, which comes in the IBM Linux on Power repository extra toolset.  I then ran it but get many 100's of lines of output - too much to paste here but the answer was hidden away in the data. Here is the section for the first Mellanox adapter: Note the Device name is in the lsvpd output but not the Vendor (Mellanox), so the lspci output saved the day:

*TM MT27520 Family [ConnectX-3 Pro]
*CD 10141007
*YC 0
*YL U78CB.001.WZS0FW8-P1-C5
*FC
*DS Ethernet PCI Adapter
*AX enP2p1s0
*NA e41d2d2d0840
*RM 2.36.5000
*YL U78CB.001.WZS0FW8-P1-C5
*FC
*DS Ethernet PCI Adapter

. . .

*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C5-T1
*FC
*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C5-T2
Key
  1. MT27520 Family = The Mellanox adapter
  2. U78CB.001.WZS0FW8-P1-C5 - Slot C5 at the back of the Server
  3. enP2p1s0 - This is the a Ethernet device name
  4. The U78CB.001.WZS0FW8-P1-C5-T1 and U78CB.001.WZS0FW8-P1-C5-T2 show there are two 10 Gb ports on the adapter

many Ethernet adapters have more ports.  The Second Mellanox adapter looks like this:

*TM MT27520 Family [ConnectX-3 Pro]
*CD 10141007
*YC 0
*YL U78CB.001.WZS0FW8-P1-C2
*FC
*DS Ethernet PCI Adapter
*AX enP3p1s0
*NA e41d2d2d1730
*RM 2.36.5000
*YL U78CB.001.WZS0FW8-P1-C2
*FC
*DS Ethernet PCI Adapter
*AX enP3p1s0d1
. . .
*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C2-T1
*FC
*DS 10GbE RoCE SFP+ PCIe 3.0 Adapter
*AX ethernet
*TM network
*YL U78CB.001.WZS0FW8-P1-C2-T2 

 

I hope this helps and stops you having to use the "empirical method" = try every adapter and every port at least twice until it suddenly blurts in to life!!

KelleyCook commented July 6 2017 lsslot -c pci is very useful too:
[root@gssio1 ~]# lsslot -c pci
# Slot Description Device(s)
U78CB.001.WZS02GN-P1-C2 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected 000a:01:00.0
U78CB.001.WZS02GN-P1-C3 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 000b:01:00.0
U78CB.001.WZS02GN-P1-C5 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 0009:01:00.0
U78CB.001.WZS02GN-P1-C6 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 0004:01:00.0
U78CB.001.WZS02GN-P1-C7 PCI-E capable, Rev 3, 16x lanes with 16x lanes connected 0000:01:00.0
U78CB.001.WZS02GN-P1-C9 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected Empty
U78CB.001.WZS02GN-P1-C10 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected 0007:80:00.0
0007:80:00.1
0007:80:00.2
0007:80:00.3
U78CB.001.WZS02GN-P1-C11 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected 0001:50:00.0
U78CB.001.WZS02GN-P1-C12 PCI-E capable, Rev 3, 8x lanes with 8x lanes connected Empty
[root@gssio1 ~]#

Additional Information

Other places to find content from Nigel Griffiths IBM (retired)

Document Location

Worldwide

[{"Line of Business":{"code":"","label":""},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"HW1W1","label":"Power -\u003EPowerLinux"},"ARM Category":[{"code":"","label":""}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
14 June 2023

UID

ibm11115631

Page Feedback