Setting up the Distributed Data Server for z/OS

Applications that want to access sysplex-wide performance data, can retrieve their input from a single data server on one system in the sysplex, which gathers the data distributed on all systems in the sysplex. Therefore, this is called the Distributed Data Server (DDS).

The DDS offers an HTTP API which can access short-term data from the Monitor III as well as long-term data from the Postprocessor. An application program can send an HTTP request for selected performance data to the DDS.

Note: Equivalent to the DDS component GPMSERVE, which gathers performance data from z/OS® systems in a sysplex, RMF XP provides a second DDS component called GPM4CIM, which you can use to gather performance data from AIX® and Linux® operating systems. For general and setup information about RMF XP refer to Cross platform monitoring with RMF XP. z/OS RMF Reporter Programmer's Guide provides information about how to exploit the HTTP API of GPM4CIM if you want to submit requests for AIX and Linux performance data. The remaining information in this topic is about GPMSERVE for z/OS only.

Exploiters of Monitor III performance data provided by the DDS are, among others, z/OS Capacity Provisioning, z/OSMF, or RMF PM. If you want to monitor systems in a sysplex, you must set up a Distributed Data Server (DDS) host session on the system in the sysplex with the highest RMF release. If you want to monitor several sysplexes, each one needs to have an active DDS.

To start the DDS, RMF provides the cataloged procedure stored in SYS1.PROCLIB(GPMSERVE):

//GPMSERVE PROC MEMBER=00
//STEP1    EXEC PGM=GPMDDSRV,REGION=256M,TIME=1440,
//         PARM='TRAP(ON)/&MEMBER'
…
//GPMPPJCL DD   DISP=SHR,DSN=SYS1.SERBPWSV(GPMPPJCL)
…

You can modify it according to your requirements.

Prerequisites for exploiting the Monitor III HTTP API

On those systems where you want to monitor short-term Monitor III data, you need to start the Monitor III gatherer with identical MINTIME and SYNC options (see Description of Monitor III data gatherer options in z/OS RMF Data Gatherer User's Guide).

Also make sure, that the following prerequisites are met on your z/OS host:

Unix System Services must be configured in full function mode.
TCP/IP under Unix System Services must be configured and fully initialized.

Prerequisites for exploiting the Postprocessor HTTP API

To get access to Postprocessor data provided by the DDS, the GPMSERVE started task points to a Postprocessor job called GPMPPJCL. A JCL template for this job is stored in SYS1.SERBPWSV(GPMPPJCL).

You must adapt or replace the GPMPPJCL member to suit your installation, ensuring that the DDS is able to run RMF Postprocessor jobs. If you do not want to request Postprocessor data with the DDS, you can omit the GPMPPJCL DD card from the GPMSERVE started task.

By default, the RMF Postprocessor retrieves data from all available intervals in the SMF buffer. You can modify the GPMPPJCL template to retrieve only SMF data from the most current interval by adding a job step similar to the following GETSMF sample job step:

//GETSMF EXEC PGM=ERBAPPL,PARM='?/*/70:78'
//SMFDATA DD DISP=(NEW,PASS),UNIT=SYSDA,SPACE=(CYL,(2,2))
//ERBLIST DD SYSOUT=*

The example job step retrieves data from SMF record types 70 through 78 for the most current interval, indicated by the question mark used for the start time. If you use ERBAPPL in your GPMPPJCL job, you must add an MFPINPUT DD card to the Postprocessor job step RMFPP in GPMPPJCL with a backward reference to the SMFDATA DD card of the GETSMF job step, for example:

//MFPINPUT DD DISP=(OLD,PASS),DSN=*.GETSMF.SMFDATA

GPMPPJCL template

/*JOBPARM  SYSAFF=*
//RMFPP    EXEC PGM=ERBRMFPP
//MFPMSGDS DD   SYSOUT=*
//XPRPTS   DD   SYSOUT=*
//XPXSRPTS DD   SYSOUT=*
//XPOVWRPT DD   SYSOUT=*
//SYSOUT   DD   SYSOUT=*
//SYSIN    DD   *

Note that the Postprocessor API functionality is only available with JES2 installed. Omit the GPMPPJCL ddname in a JES3 environment.

The complete DDS HTTP API is described in z/OS RMF Reporter Programmer's Guide.

Prerequisites for exploiting the SMF data set support of the Postprocessor HTTP API

To get access to preallocated SMF data from data sets or log streams with the Postprocessor HTTP API provided by the DDS, the GPMSERVE started task must point to a Postprocessor job with its GPMPPJCL DD card. The Postprocessor JCL provided here should be derived from the JCL template stored in SYS1.SERBPWSV(GPMPPJV2).

You must adapt this GPMPPJV2 member to suit your installation (like with GPMPPJCL) and additionally, you must provide a user ID for the USER parameter of the JOB card. The user ID used at this place depends on the setting of the HTTP_NOAUTH option in your GPMSERVE started task.

Distributed Data Server running with HTTP_NOAUTH(*) or HTTP_NOAUTH(ip-addr):
Provide a universal user ID, GPMUID, with the USER parameter on the JOB statement (USER=GPMUID); here, universal means that the user ID can be used by all exploiters of the DDS (by all authorized exploiters in the case of HTTP_NOAUTH(ip-addr) to access the SMF data sets. The user ID GPMUID may be empty, in which case the Postprocessor job runs under the authority of the GPMSERVE started task user ID.
Distributed Data Server running with HTTP_NOAUTH():
Leave the //*UID line in the GPMPPJV2 template unchanged in this case. The Distributed Data Server takes the user ID which is used for authorization to the DDS to substitute the user ID in the USER parameter.

The GPMSERVE user ID needs permission to submit jobs on behalf of GPMUID and it needs UPDATE access to the job output on the JES spool. The following example shows RACF® commands which allow these actions:

RDEFINE SURROGAT GPMUID.SUBMIT UACC(NONE) OWNER(GPMUID)
PERMIT GPMUID.SUBMIT CLASS(SURROGAT) ID(GPMSERVE) ACCESS(READ)
SETROPTS RACLIST(SURROGAT) REFRESH

RDEFINE JESSPOOL nodename.GPMUID.GPMSERVE.** UACC(NONE) OWNER(GPMUID)
PERMIT nodename.GPMUID.GPMSERVE.** CLASS(JESSPOOL) ID(GPMSERVE) ACCESS(UPDATE)
SETROPTS RACLIST(JESSPOOL) REFRESH

In order to use the job deletion functionality implemented with the joboutdel request parameter, a RACF profile in the JESJOBS class has to be defined like in the following example for the user GPMUID and the Postprocessor job GPMDDSPP started by GPMSERVE:

RDEFINE JESJOBS PURGE.nodename.GPMUID.GPMDDSPP UACC(NONE)
PERMIT PURGE.nodename.GPMUID.GPMDDSPP CLASS(JESJOBS) ID(GPMUID) ACC(ALTER)

Setting up secure communication for the Distributed Data Server

This topic describes how to enable secure communication with the DDS using Application Transparent - Transport Layer Security (AT-TLS).

Note: Using AT-TLS to secure communication with the DDS affects all DDS clients. If AT-TLS is used to enable secure communication with the DDS, all DDS clients, e.g. the RMF CIM provider, have to use secure communication with the DDS. Unsecure communication with the DDS is no longer possible and will fail.

Before you can start with the DDS setup, you must enable the Policy Agent for AT-TLS. Information about setting up AT-TLS communication is provided in z/OS Communications Server: IP Configuration Guide.

The example in Figure 1 uses a RACF key ring. The key ring must contain a valid default certificate Start of change , otherwise the certificate label to be used must be specified using the TTLSEnvironmentAdvancedParms ServerCertificateLabel End of change . The DDS server started task user ID (e.g. GPMSERVE) must have access to the key ring. Consider using a certificate from a trusted Certificate Authority (CA), since this may be required by RMF DDS Clients.

After you have completed your Policy Agent configuration, use the REFRESH command to refresh the Policy Agent configuration:

MODIFY PAGENT,REFRESH

For more information, see z/OS Security Server RACF Security Administrator's Guide. For other security management products, refer to your security product documentation.

RMF Distributed Data Server AT-TLS rule

Start of change The following example shows a rule that enables secure communication with the DDS. End of change

TTLSRule: Jobname: The name value specifies the job name of the application. GPMSERVE is the job name of the DDS.
TTLSRule: LocalPortRange: The local port the application is bound to for this rule's action to be performed. 8803 is the default HTTP Port of the DDS.
TTLSRule: Direction: Specifies the direction the connection must be initiated from for this rule's action to be performed. In this example, Inbound is specified, which means that the rule applies to connection requests that arrive to the local host.
TTLSRule: Priority: An integer value in the range 1 - 2000000000 that represents the priority associated with the rule. The highest priority value is 2000000000.
When using multiple rules for the DDS server, the more specific the rule, the higher its priority should be. Generic rules without detailed specifications of the incoming connections should have a low priority.
TTLSEnvironmentAction: HandshakeRole: Specifies the SSL handshake role to be taken for connections in this AT-TLS environment. In this example, Server is specified which means that the SSL handshake is performed as a sever.
TTLSKeyringParms: Keyring: Specifies the path and file name of the key database z/OS® UNIX file, the ring name of the SAF key ring, or the name of the z/OS PKCS #11 token. In this example, the RACF key ring DDSServerKeyring is specified.
TTLSEnvironmentAdvancedParms: ServerCertificateLabel: Specifies the label of the certificate for a server application to authenticate the server. In this example, the DDS server certificate with the label RMFDDS is used.
TTLSEnvironmentAdvancedParms: SSLv2: Specifies the state of the SSL Version 2 protocol. In this example, Off is specified, which means that the SSL Version 2 protocol is disabled.
TTLSEnvironmentAdvancedParms: SSLv3: Specifies the state of the SSL Version 3 protocol. In this example, Off is specified, which means that the SSL Version 3 protocol is disabled.
TTLSEnvironmentAdvancedParms: TLSv1: Specifies the state of the TLS Version 1 protocol. In this example, Off is specified, which means that the TLS Version 1 protocol is disabled.
TTLSEnvironmentAdvancedParms: TLSv1.1: Specifies the state of the TLS Version 1.1 protocol. In this example, Off is specified, which means that the TLS Version 1.1 protocol is disabled.
TTLSEnvironmentAdvancedParms: TLSv1.2: Specifies the state of the TLS Version 1.2 protocol. In this example, On is specified, which means that the TLS Version 1.2 protocol is enabled.

AT-TLS rule for clients, using secure communication with the RMF Distributed Data Server

The following example shows a rule to enable a client running on z/OS to establish secure communication with the DDS. The RACF key ring DDSClientKeyring must contain the server certificate that is used by the RMF Distributed Data Server rule.

When using a server certificate from a trusted certificate authority (CA), the virtual RACF CA key ring *AUTH*/* can be used.

Figure 2. Sample Policy Agent rule for secure communication for DDS clients

# RMF Distributed Data Server Client Rule
TTLSRule                          DDSClientRule
{
   RemotePortRange                8803
   RemoteAddr                     1.0.0.0
   Direction                      Outbound
   TTLSGroupActionRef             DDSClientGRP
   TTLSEnvironmentActionRef       DDSClientENV
}
TTLSGroupAction                   DDSClientGRP
{
   TTLSEnabled                    On
   Trace                          1
}
TTLSEnvironmentAction             DDSClientENV
{
   HandshakeRole                  Client
   TTLSKeyringParms
   {
      Keyring                     DDSClientKeyring
   }
}

TTLSRule: RemoteAddr: The remote IP address specification that must match for this rule's action to be performed—that is, the IP address of the DDS to which the client is connecting.
TTLSRule: RemotePortRange: The remote port to which the application must be connecting for this rule's action to be performed. The default HTTP port of the DDS is 8803.
TTLSRule: Direction: Specifies the direction from which the connection must be initiated to perform this rule’s action. In this example, Outbound is specified, which means that the rule applies to connection requests that are initiated by the application.
TTLSEnvironmentAction: HandshakeRole: Specifies the SSL handshake role to be taken for connections in this AT-TLS environment. In this example, Client is specified, which means that the SSL handshake is performed as a client.
TTLSKeyringParms: Keyring: Specifies the z/OS UNIX path and file name of the key database file, the ring name of the SAF key ring, or the name of the z/OS PKCS #11 token. In this example, the RACF key ring DDSClientKeyring is specified.

RMF Distributed Data Server AT-TLS rule with client certificate authentication

The DDS supports the communication scenario in which a client authenticates itself via a client certificate that is associated with a RACF user ID. No user ID/password authentication is performed in this case. To enable this communication mode, the CLIENT_CERT(ACCEPT) option must be set in the DDS parmlib member.

The following example shows a rule that uses a RACF key ring. The RACF key ring DDSServerKeyring must contain the server certificate that is used by the rule. The server certificate must either be the default certificate, or the certificate label to be used must be specified using the TTLSEnvironmentAdvancedParms ServerCertificateLabel. The DDS server started task user ID (for instance, GPMSERVE) must have access to the key ring. Consider using a certificate from a trusted certificate authority (CA), since this may be required by RMF DDS clients.

The DDSServerKeyring must also contain the client certificate associated with a RACF user ID. If your access to Monitor III data is protected by RACF resource profile ERBSDS.MON3DATA, the associated user ID must have READ access.

For more information, see z/OS Security Server RACF Security Administrator's Guide.

Figure 3. Sample Policy Agent rule for secure communication with client certificate authentication for the DDS

# RMF Distributed Data Server Rule
TTLSRule                          DDSServerClientCertRule
{
   LocalPortRange                 8803
   Jobname                        GPMSERVE
   RemoteAddr                     1.0.0.0
   Direction                      Inbound
   TTLSGroupActionRef             DDSServerClientCertGRP
   TTLSEnvironmentActionRef       DDSServerClientCertENV
   Priority                       2
}
TTLSGroupAction                   DDSServerClientCertGRP
{
   TTLSEnabled                    On
   Trace                          1
}
TTLSEnvironmentAction             DDSServerClientCertENV
{
   HandshakeRole                  ServerWithClientAuth
   TTLSKeyringParms 
   {
      Keyring                     DDSServerKeyring
   }
   TTLSEnvironmentAdvancedParms             
   {                                        
      ClientAuthType              SAFCheck
      ServerCertificateLabel      RMFDDS
   }
}

TTLSRule: Jobname: The name value specifies the job name of the application. GPMSERVE is the job name of the DDS.
TTLSRule: LocalPortRange: The local port to which the application is bound for this rule's action to be performed. The default HTTP port of the DDS is 8803.
TTLSRule: RemoteAddr: The remote IP address specification that must match for this rule's action to be performed. You can use this option to specify that this rule applies only to a subset of client connections, where the clients are running on a specific system. When using multiple rules for the DDS server, be sure to use the TTLSRule Priority setting.
TTLSRule: Priority: An integer value in the range 1 - 2000000000 that represent the priority associated with the rule. The highest priority value is 2000000000.
When using multiple rules for the DDS server, the more specific a rule, the higher its priority should be.
TTLSRule: Direction: Specifies the direction from which the connection must be initiated for this rule's action to be performed. In this example, Inbound is specified, which means that the rule applies to connection requests that arrive at the local host.
TTLSEnvironmentAction: HandshakeRole: Specifies the SSL handshake role to be taken for connections in this AT-TLS environment. In this example, ServerWithClientAuth is specified, which means that the SSL handshake is performed as a sever, and a client certificate is required.
TTLSKeyringParms: Keyring: Specifies the z/OS UNIX path and file name of the key database file, the ring name of the SAF key ring, or the name of the z/OS PKCS #11 token. In this example, the RACF key ring DDSServerKeyring is specified.
TTLSEnvironmentAdvancedParms: ClientAuthType: Specifies the type of client certificate validation to be performed for connections in this AT-TLS environment. Client certificates are requested only if HandshakeRole is set to ServerWithClientAuth. That option needs to be set to SAFCheck, which requires the client to present a certificate, to perform a client certificate validation and to have a client certificate which is associated with an user ID defined to the security product. If the option is not set to SAFCheck, the DDS will not use client certificate authentication.
TTLSEnvironmentAdvancedParms: ServerCertificateLabel: Specifies the label of the certificate for a server application to authenticate the server. In this example, the DDS Server certificate with the label RMFDDS is used.

AT-TLS for clients, using secure client certificate authentication with the RMF Distributed Data Server

The following example shows a rule to enable a client running on z/OS to establish secure communication to the DDS with client certificate authentication. No user ID/password authentication is performed. The RACF key ring DDSClientKeyring must contain the server certificate used by the RMF Distributed Data Server rule and the client certificate. The client certificate must either be the default certificate, or the certificate label to be used must be specified using the TTLSEnvironmentAdvancedParms CertificateLabel. Also, the client certificate must be associated with a RACF user ID. If your access to Monitor III data is protected via RACF resource profile ERBSDS.MON3DATA, the associated user ID must have READ access.

Figure 4. Sample Policy Agent rule for secure communication for DDS clients with client certificate authentication

# RMF Distributed Data Server Client Rule for Client Certificate Authentication
TTLSRule                           DDSClientCertRule
{
   RemotePortRange                 8803
   RemoteAddr                      1.0.0.0
   Direction                       Outbound
   TTLSGroupActionRef              DDSClientCertGRP
   TTLSEnvironmentActionRef        DDSClientCertENV
}
TTLSGroupAction                    DDSClientCertGRP
{
   TTLSEnabled                     On
   Trace                           1
}
TTLSEnvironmentAction              DDSClientCertENV
{
   HandshakeRole                   Client
   TTLSKeyringParms
   {
      Keyring                      DDSClientKeyring
   }
   TTLSEnvironmentAdvancedParms 
   {
      CertificateLabel             RMFDDSCLIENT
   }
}

TTLSRule: RemotePortRange: The remote port to which the application must be connecting for this rule's action to be performed. The default HTTP port of the DDS is 8803.
TTLSRule: RemoteAddr: The remote IP address specification that must match for this rule's action to be performed—that is, the IP address of the DDS to which the client is connecting.
TTLSRule: Direction: Specifies the direction from which the connection must be initiated for this rule's action to be performed. In this example, Outbound is specified, which means that the rule applies to connection requests that are initiated by the application.
TTLSEnvironmentAction: HandshakeRole: Specifies the SSL handshake role to be taken for connections in this AT-TLS environment. In this example, Client is specified, which means that the SSL handshake is performed as a client.
TTLSKeyringParms: Keyring: Specifies the z/OS UNIX path and file name of the key database file, the ring name of the SAF key ring, or the name of the z/OS PKCS #11 token. In this example, the RACF key ring DDSClientKeyring is specified.
TTLSEnvironmentAdvancedParms: CertificateLabel: Specifies the label of the certificate to be used for authentication. In this example, the DDS client certificate with the label RMFDDSCLIENT is used.

Prerequisites for IBM® Health Checker for z/OS®

Health checks for RMF are available in the IBM Health Checker. To define health checks and to read the Policy Agent definitions, the GPMSERVE user ID needs permission. The following example shows RACF commands that allow these actions:

RDEFINE XFACILIT HZS.<system|*>.IBMRMF.*.ADD UACC(NONE)
PERMIT HZS.<system|*>.IBMRMF.*.ADD CLASS(XFACILIT) ID(GPMSERVE) ACCESS(CONTROL)
SETROPTS RACLIST(XFACILIT) REFRESH

RDEFINE SERVAUTH EZB.PAGENT.<system|*>.<tcpip stack|*>.TTLS UACC(NONE)
PERMIT EZB.PAGENT.<system|*>.<tcpip stack|*>.TTLS CLASS(SERVAUTH) ID(GPMSERVE) ACCESS(READ)
SETROPTS RACLIST(SERVAUTH) REFRESH

For more information about the health checks available for RMF, see RMF checks (IBMRMF) in IBM Health Checker for z/OS User's Guide.