IBM Support

Configuring Shared Memory Communications Version over RDMA version 2 (SMC-Rv2) For Your z/OS Environment

How To


Summary

This document provides an overview on how to properly implement SMC-Rv2 on a z/OS environment.

Objective

This document includes:
  • SMC-Rv2 requirements
  • TCP/IP profile configuration

Environment

Hardware Requirements
  • IBM z15 or z16
  • RDMA over Converged Ethernet (RoCE) Express2 or RoCE Express3*
    • PNetIDs assigned
      • OSA and RoCE PNetIDs must be configured and match
    • PFIDs defined 
Software Requirements
  • Software: z/OS 
    • IBM z/OS V2R5
    • Define at least one user-defined Enterprise ID in z/OS! (disabled by default)
  • Environments: LPAR or z/VM guests
A diagram showing two LPARS with a TCP data connection between ISM devices on each partner

Steps

Setting up the environment for Shared Memory Communications over RDMA
  1. Install and configure the “RoCE Express2” or “RoCE Express3” (z16) features in the hardware configuration definition (HCD).
  2. Assign PNetID values and configure the values in the HCD for both the “RoCE Express” ports and any OSA-Express devices that use the “RoCE Express2” or “RoCE Express3” ports for SMC-R communications.
  3. Configure Ethernet switches for RoCE Express features.
    • Switch port must support jumbo frames if RoCE MTU size is configured at 2K or 4K
    • Switch port must be configured in TRUNK mode and have all OSA VLAN in the access list.
Hardware Configuration Definition
On the HCD main panel, do the following:
  1. On the Primary Task Selection page window, select “Define configuration data”.
    • The z/OS TN3270 window showing the Hardware Configuration Primary Task Selection panel

  2. On the resulting panel, select “Processors”. 

    • "define modify or view configuration data" panel of HCD

  3. On the Processor List panel, select an eligible processor and action “Work with PCIe functions”.

    • Processor List of HCD

  4. Use PF20=Right to scroll to the partition assignments for the displayed PCIe functions, one panel for each defined channel subsystem

    • RoCE Adapter defined FID used by TCPIP configPCIe function list of HCD

  5. Use F11=Add to define a new PCIe function

    • HCD panel showing Add PCIe Function window

  6. Set PCIe Adapter Type to ROCE-2

    • Add adapter type window listing options which includes ISM and RoCE-2

  7. Function ID should match TCP/IP INTERFACE statement (for SMC-Rv2)

    • HCD panel showing Add PCIe function window

  8. Associate the RoCE Express Adapter to an OSA interface with same PNetID
  •  Each physical network ID entry correlates to a physical port on the adapter.Add/modify physical network IDs window in HCD

Sample RoCE IOCP / IOCDS declaration

​​​​Sample RoCE IOCP / IOCDS declaration for SMC-Rv2 device

z/OS TCP/IP profile configuration requirements

  1. Configure GLOBALCONFIG SMCR statement.
  2. Add EID on SMCGLOBAL statement.
  3. Define SMCR, RoCEv2 IP address (SMCRIPADDR) and PFID (configured on OSA INTERFACE statement).

Start the interfaces (RNIC is automatically activated).

Here are the format of the definitions needed for the TCP/IP profile:

; ----------------------------------------------------------------------

; GLOBALCONFIG statements to enable SMC-Rv2

; ----------------------------------------------------------------------

GLOBALCONFIG SMCR  

  SMCGLOBAL SMCEID user_EID_values ENDSMCEID

; ----------------------------------------------------------------------

; INTERFACE statements for IPv4

; ----------------------------------------------------------------------

; For OSA-Express QDIO ethernet

INTERFACE intf_name 

  DEFINE IPAQENET

  CHPIDTYPE OSD

  PORTNAME port_name

  IPADDR ipv4addr/num_mask_bits

  SMCR PFID pfid

  SMCRIPADDR ipv4addr

  SMRMTU mtusize                    ; optional

Additional Information

Verification

z/OS TCP/IP profile configuration SYNTAXCHECK

  • Can be used to validate SMCR profile changes before use
  • Run against profile data set

Format:

V TCPIP,procname,SYNTAXCHECK,datasetname

Example:

V TCPIP,TCPIPB,SYNTAXCHECK,DSN=SYS1.TCPPARMS(PROFSYSB)
EZZ0060I PROCESSING COMMAND: VARY TCPIP,TCPIPB,SYNTAXCHECK,DSN=SYS1.TCPP
 ARMS(PROFSYSB)
 EZZ0061I VARY SYNTAXCHECK COMMAND BEGINNING
...
 EZZ0316I PROFILE PROCESSING COMPLETE FOR FILE 'SYS1.TCPPARMS(PROFSYSB)'
 EZZ0062I VARY SYNTAXCHECK FOUND NO ERRORS
 EZZ0065I VARY SYNTAXCHECK COMMAND COMPLETE

SMC-R Monitoring - DISPLAY PCIE 

  • Activation of first SMC-R capable OSD causes PFIDs to be activated
  • Use DISPLAY PCIE to display defined PFIDs

Format:

D PCIE

Example:

D PCIE

IQP022I 19.06.25 DISPLAY PCIE 545
PCIE 0013 ACTIVE
 

PFID DEVICE TYPE NAME STATUS ASID JOBNAME CHID VFN PN
00000111 ISM ALLC 002A NET 07C2 0001
00000011 RoCE Express3 ALLC 002A NET 01D8 0001 1
00000021 RoCE Express3 ALLC 01BC 0001 1


If the devices are in standby, issue a system command "CF PFID(pfid),ONLINE".

Netstat Devlinks/-d for OSD report

  • SMC V1 and V2 capabilities
  • If V2 capable, displays configured PFID, SMCRIPADDR, and SMCRMTU

Format:

D TCPIP,procname,NETSTAT,DEVlinks

Example:

D TCPIP,TCPIPB,NETSTAT,DEVLINKS

…

IntfName: GIG3 IntfType: IPAQENET IntfStatus: Ready

  PortName: GIG2E Datapath: OBE2 DatapathStatus: Ready

  CHPIDType: OSD   SMCR: Yes

  PNetID: IBMTRUNKSET1   SMCD: Yes

...

SMCR Capability: V2

  Associated Multi-subnet RNIC interface: EZARIUT10011

  RoCE PFID: 0011 SMCRMTU: 1024

  SMCRIPADDR: 9.82.24.157

  Unassociated ISM interfaces: EZAISMU1

IntfName: EZARIUT10011  IntfType: RNIC  IntfStatus: READY

Netstat CONFIG/-f report

Displays:
  • User-defined EIDs
  • SMC exclude and permit lists

Format:

DTCPIP,procname,NETSTAT,CONFIG

Example:

D TCPIP,TCPIPB,NETSTAT,CONFIG

…

Global Configuration Information:

SMCGlobal:

AutoCache: YES AutoSMC: YES

SMCEID: IBM-COMMSERVER-MULTIPASS

SMC Exclude List: NONE

SMC Permit List: ALL

SMCR: YES

FixedMemory: 0000256M TcpKeepMinInt: 00000300

SMCD: YES

FixedMemory: 0000256M TcpKeepMinInt: 00000300

SMCDSYSEID: YES (IBM-SYSZ-ISMSEID0000000023B83931)

Netstat ALL/-a report

  • Displays SMC version and EID
Example:

D TCPIP,TCPIPD,N,ALL,IPPORT=9.82.24.151+21 

EZD0101I NETSTAT CS V2R4 TCPIPD 331                      

CLIENT NAME: FTPSERVE                 CLIENT ID: 00000CC2 

  LOCAL SOCKET: ::FFFF:9.82.24.153..21                    

  FOREIGN SOCKET: ::FFFF:9.82.24.151..1242 

…

 SMC INFORMATION:                                                

       SMCDSTATUS:       ACTIVE           SMCDVERSION:        2      

         LOCALSMCLINKID: 02010001         REMOTESMCLINKID:    01010001

         LOCALSMCRCVBUF: 64K              REMOTESMCRCVBUF:    64K    

         SMCEID:         IBM-SYSZ-ISMSEID0000000023B83931             

For more information, see resources below.

Document Location

Worldwide

Operating System

z/OS:All operating systems listed

[{"Type":"MASTER","Line of Business":{"code":"LOB56","label":"Z HW"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG90","label":"z\/OS"},"ARM Category":[{"code":"a8m0z0000000AfAAAU","label":"z\/OS Comm Server-\u003ETCPIP-\u003EIF\/DLC - Data Link Control-\u003ESMC-R - Shared Memory Comm. under RDMA"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Product Synonym

Z; Z systems; z/OS; Communications Server

Document Information

Modified date:
02 May 2023

UID

ibm16985517

Page Feedback