Configuration Instructions for Electronic Customer Support (ECS) and Electronic Service Agent (ESA) for IBM i

Resolving The Problem

• Firewall Requirement

  The following is a summary from the Electronic Service Agent (ESA) and Electronic Customer Support (ECS) VPN and HTTP Firewall Settings document:
  
  V7R3 and higher, ECS/ESA use EDGE server that use esupport.ibm.com destination host name that needs to be allowed on Firewall for 443.
  V7R2 uses LEGACY servers to connect for ECS/ESA. By replacing a configuration file, and specific PTFs, it's possible to use the EDGE server, which is recommended. Information on Firewall document.
  
  For HTTP/HTTPS communication, ports 80 and 443 need to be open for both outbound and inbound (return) TCP traffic. 

  Summary of actions required for incoming changes:

  Enable firewall access to IP addresses before 1 March 2024:

  • 192.148.6.11 (port 443)
  • 170.225.123.67 (port 443)
  • 2620:1f7:c010:1:1:1:1:11 (port 443)  - if IPv6 connections required

  Enable DNS or a firewall/proxy connectivity solution in place of static IP addresses before 1 November 2024.
   

  Considerations:

  • Typically, if a firewall port is opened for outbound communication, return traffic is automatically allowed on that port.
  • For 7.3, having PTF SI68172 on the system would require ONLY port 443 to be open. Port 80 is no longer needed. For 7.2, having PTF SI76398 on the system do the same. ONLY VALID WHEN USING EDGE CONFIGURATION.
  • ESA has its own internal certificate to exchange with the IBM backend server, so any 'addition' by Proxy/Firewall during the communication makes it fail. If the environment has a Proxy/Firewall that is terminating the SSL connection and returning its own self-signed certificate, it is not supported. This also can happen if communication is being inspected.
  • Infrastructure improvements to electronic fix distribution were implemented. IP and hostnames are changing for servers that support fix delivery. This affects both LEGACY and EDGE servers. More information on the Firewall document.

• IBM i OS Prerequisites

  Ensure that the following prerequisites are satisfied before creating the service configuration and testing:

  1. Product options:

  The following product options must be installed on the system. In GO LICPGM menu, select option 10. Display installed licensed programs and press F11 twice to see Product Option:

  • 5770SS1 Option 30 Qshell
  • 5770SS1 Option 33 Portable Application Solutions Environment (PASE)
  • 5770SS1 Option 34 Digital Certificate Manager
  • 5770JV1 Option 16 Java SE 8 32 bit
  • 5770JV1 Option 17 Java SE 8 64 bit
  • 5733SC1 *BASE - IBM Portable Utilities for i 
  • 5733SC1 Option 1 - OpenSSH, OpenSSL, zlib 
  • 5770UME IBM Universal Manageability Enablement for i 
     

  Review the Standard and Keyed Media Set - Master List for information on where to find any missing product.

  
  Note: ECS/ESA need JDK8 (JV1 options 16 & 17) to be on the system, plus recommended PTFs to use that Java version. Information could be found in the Migrate Electronic Service Agent (ESA) from using Java 6 to Java 8 document.

  
  Important information about 5770UME and the *CIMOM Server Associated with the Product:

  Product 5770UME (IBM Universal Manageability Enablement for i), IS required for Electronic Service Agent. Specially, starting at 7.2 to start Electronic Service Agent jobs at IPL for Call Home and to collect Inventory. There are several hardware inventory types that are not collected when this product is not installed, and in some cases, will not activate. The CIMOM server associated with this product (QUMECIMOM) does need to be started for ECS/ESA functions to be successful.

  It also requires a certificate assigned to CIMOM server. Additional information about CIMOM server certificate and creation steps could be found on the *CIMOM server certificate document.

  It is recommended that the latest PTFs for the 5770UME product are applied on the system.

  2. The Retain server security data (QRETSVRSEC) system value must be set to 1:

  • CHGSYSVAL SYSVAL(QRETSVRSEC) VALUE(1)

  3. TCP configuration:

  TCP needs to be configured correctly in order to avoid communication problems. In the CFGTCP menu:

  • Option 1. Work with TCP/IP interfaces.

  IP address of the system has to be ACTIVE. Same for LOOPBACK interface.

  Use PING LOOPBACK and PING LOCALHOST to ensure it is successful. Both should return 127.0.0.1.

  • Option 2. Work with TCP/IP routes.

  There has to be a *DFTROUTE. The Next Hop has to be valid as well.

  • Option 12. Change TCP/IP domain information.

  Must have a host name and domain name.
  The Host Name Search Priority should be set to *LOCAL
  There should be at least one IP address listed under the Domain Name Server. If there are multiple addresses, each must be unique.

  • Option 10. Work with TCP/IP host table entries.

  IP address of the system (from option 1) has to be associated with the long name of the system (HostName.DomainName from option 12), and the short name for the system (Host Name from option 12)
  Long name should be listed first, followed by a short name. There should not be any other Host Table Entries that have the same short and/or long name for the system.

  4. Host Servers:
  Host Servers needs to be running.

  • Check whether these ports, 8470-8476, can be used. More information in following link:

  Port numbers for host servers and server mapper

  Review NETSTAT Option 3 to verify that Host Servers are started or restart them by doing:
  ENDHOSTSVR *ALL

  STRHOSTSVR *ALL

  Are the host servers started? If started, then try the following commands:

  STRQSH

  export -s CLASSPATH=/QIBM/ProdData/HTTP/Public/jt400/lib/jt400.jar

  java utilities.JPing localhost

  This checks if toolbox whether able to connect to the local Host Servers. Something like this display if all is correct:

  > java utilities.JPing localhost                              
  
    Verifying connections to system localhost...                
                                                                
    Successfully connected to server application:  as-file      
    Successfully connected to server application:  as-netprt    
    Successfully connected to server application:  as-rmtcmd    
    Successfully connected to server application:  as-dtaq      
    Successfully connected to server application:  as-database  
    Successfully connected to server application:  as-ddm       
    Successfully connected to server application:  as-central   
    Successfully connected to server application:  as-signon    
    Connection verified                                         
    $ 

• Electronic Services recommended PTFs

  The latest Electronic Services Group PTF:

  SF99627 for 7.1
   

  The latest Recommended Fixes for Electronic Services:

  • IBM i 7.1 Recommended Fixes - Electronic Services
  • IBM i 7.2 Recommended Fixes - Electronic Services
  • IBM i 7.3 Recommended Fixes - Electronic Services
  • IBM i 7.4 Recommended Fixes - Electronic Services
  • IBM i 7.5 Recommended Fixes - Electronic Services
     

• Update Contact information

  To verify or change the system contact information:

  WRKCNTINF

  Select option 2. Work with local service information

  Select option 1. Display service contact information

  If changes are needed, select option 2. Change service contact information

  
  Verify the following:

  • The Company and Contact Name are correct.
  • The Customer number: Customer identifier and Customer description and the Contract number: Contract identifier and Contract description are all be set to *NONE.
  • The Primary contact telephone number and/or Help desk or pager number are correct.
  • The Fax telephone number is correct, if applicable.
  • The correct address is listed for the Mailing address, and that the City or locality, State or province, Country or region and Postal code are on the designated lines.
    Note: The Country or Region should be the two-character abbreviation for the country or region, such as US for United States, CA for Canada, SE for Sweden.
    Two-digit country code can be found here: Country or Region Codes

  • A Primary email address is specified.

  • The Media for PTFs is set to *DVDROM. 

  Note: If Call central site support is set to *YES, the Help desk or pager number will be used as the primary contact telephone number.
  
  Due to the changes introduced in the New Support Platform in order to receive notification when doing CALL HOME TEST, the Contact Information should match an existing IBMid.
  Additional information is found in Call Home Contacts: Guideline and Best Practices Documentation for Clients

• Delete and re-create the Service Configuration

  The instructions below are intended to delete any previous service configuration, Point-To-Point profiles, line descriptions and configuration files that remain and re-create it.
  Include basic steps to force Contact Information change and ESA enablement.
  
  Reference to the CRTSRVCFG Command document for other configuration options and additional information.

  Do the following while signed on with a user profile that has *SECOFR authority:

  • CHGSRVAGTA ENABLE(*NO)
  • DLTSRVCFG DLTCMNCFG(*YES)
  • WRKTCPPTP and delete (4=Remove) any remaining profiles that begin with the letter 'Q'
  • WRKLIND LIND(Q*) and delete (4=Delete) any remaining lines that have the type of *PPP and begin with the letter 'Q'. Also, delete QESLINE, QTILINE, and Q1PLIN type *SDLC, if exist.
  • RMVLNK OBJLNK('/qibm/userdata/os400/universalconnection/*')

  Note: An error like 'Requested operation not allowed. Access problem.' might appear. Disregard.

  • RMVLNK OBJLNK('/qibm/userdata/os400/serviceagent/Contact.properties')
  • CHGCNTINF MAILADDR(*SAME *SAME *SAME *SAME *SAME XX *SAME) MEDPTF(*DVDROM)

  Where XX is the two-character abbreviation for the country or region, such as US for United States, CA for Canada, SE for Sweden.

  • CRTSRVCFG ROLE(*PRIMARY) CNNTYPE(*DIRECT) CNTRYID(*SELECT) STATE(*SELECT)

  Select the appropriate country/region and state/province (where the system is physically located) If there is a proxy server involved, specify the appropriate proxy information.

  • CHGSRVAGTA ENABLE(*YES)

  Note: If msgCPD0A35 is received after run CRTSRVCFG, check the RC related and refer to the CRTSRVCFG/CHGSRVCFG fails with msgCPD0A35: Java exception not handled for user-defined document for usual stuff that could cause the error.

• Enable Electronic Service Agent (ESA)

  The instructions are intended to clean up any old ESA scheduled job entries that might still exist or have been restored.

  Do the following while signed on with a user profile that has *SECOFR authority:

  • GO SERVICE

  Select option 1. Change Service Agent attributes

  Specify Enable as *NO and press <ENTER>.

  Press F12=Cancel

  • WRKJOBSCDE JOB(QS*)

  Select option 4=Remove next to the following entries, if they exist:

  QSJERRRPT QSJHEARTBT QS9AUTOPTF QS9AUTOTST or QS9SACOL

  • GO SERVICE

  Select option 1. Change Service Agent attributes

  Specify Enable as *YES and press <ENTER>.

  Page down to the second page

  In the Service Information section, change the Collect Time to a time when the system is least busy and press <ENTER>.

  • GO SERVICE

  Select option 10. Work with jobs

  The following jobs should be active: QS9HDWMON, QS9PALMON, QS9PRBMON, QS9PRBSND 

  Press F12=Cancel

  Note: The job QS9SFWMON won't be listed if the PTF (or superseded) for the appropriate OS version is applied. Reference to the Disable automatic problem reporting of legacy Software FFDC (First Failure Data Capture) - QS9SFWMON job document for additional information.

  Starting 7.5, the ability to report a Software problem entry was removed. Due to changes introduced in the Support Platform, the WRKPRB entries related to Software are no longer accepted when manually reported in older releases either.

  Note: QS9UAK might also be shown as a scheduled job. This job is only required when system is a Standalone partition over Power 8 or higher. Otherwise, it's not required. Could be disabled by doing CHGSRVAGTA REFRESHUAK(*NO). More information about UAK and ESA could be found in the Manage update access keys with IBM Electronic Service Agent (ESA) on IBM i document.
  The QS9AUTOPTF scheduled job is removed starting 7.5

  More information about scheduled jobs could be found in the Electronic Service Agent Jobs document.

• Testing ECS/ESA functions

  • The following command orders the latest cumulative package cover letter:

  SNDPTFORD PTFID((SF98xxx *ONLYPRD *ONLYRLS))

  Where xxx is the OS release, that is, SF98730 for 7.3, SF98740 for 7.4 and SF98750 for 7.5.

  • To test Call Home by ESA, send a test problem by using the following steps:

  GO SERVICE

  Select option 15. Send test problem.

  Leave the Error log identifier as 00000000 and press <ENTER>.

  Press F12=Cancel

  WRKPRB to monitor the status of the Test Problem

  Press F5=Refresh

  The test problem should go to a SENT status.

  • To test connectivity to IBM backend servers, use below commands.  Test plain TCP communication:

  - VFYSRVCFG SERVICE(*ECS) VFYOPT(*ALL)

  Let it complete, might take a while.
  
  - VFYSRVCFG SERVICE(*FIXREP) VFYOPT(*ALL)

  Let it complete, might take a while.
  
  - VFYSRVCFG SERVICE(*PRBRPT) VFYOPT(*ALL)

  Let it complete, might take a while.
  
  - VFYSRVCFG SERVICE(*SPCFG) VFYOPT(*ALL)

  Let it complete, might take a while.
  
  - VFYSRVCFG SERVICE(*SRVAGT) VFYOPT(*ALL)

  Let it complete, might take a while.

   

  The VFYSRVCFG commands log to joblog:  IP address, protocol, and port used along with success or failure information.

  CPIAC59: Verification was successful.
  CPIAC60: Verification was not successful.
  CPIAC61: The value does not match an existing service destination.
  Recommended PTFs for ESA and 'Delete and re-create service configuration' steps need to be followed correctly.

• Collecting data for analysis

  Following the steps described allow to configure ECS/ESA.

  If there are failures at any point in the steps, generate a job log: 
  DSPJOBLOG JOB(*) OUTPUT(*PRINT)

  Using the Support Portal, use the Search Support option to search on any error messages for resolution.

  If the problem needs to be reported to IBM Support Center, collect the information by using QMGTOOLS. Document QMGTOOLS: Collect ECS/ESA data (COLESADTA) has the following steps:

  • Assure to have last version of QMGTOOLS prior to the collection: 

  GO QMGTOOLS/MG 
  Use option 13. Check IBM for updated QMGTOOLS. 
   
  If not on the system, or manually action is needed, install or update it by using following document (Method 3 - Manually update or install from the IBM public FTP site)
  QMGTOOLS: How to check and update QMGTOOLS

  • Once on the system, collect the information by using following command: 

  QMGTOOLS/COLESADTA OUTPUT(*IFS) VFYSRVCFG(Y) SNDTOIBM(Y) PMR#(TSxxxxxxx) 
  Where TSxxxxxxx is the Case number assigned.
   

  • FTPTYPE: Defaults for *HTTPS. Note: You would need a transfer ID and password. Use the following document for instructions on how to get one QMGTOOLS: Create a transfer ID to upload data to IBM
     

  
  Instructions are meant for transfer ID already stored (*STORED). Review following document for instructions. 
  Store your IBM Support Transfer ID credentials in QMGTOOLS for use with the FTP2IBMCMD CL command.

  • If the data is not sent, look for the ESADOCS.zip file generated and manually attach to the Case. 

  MustGather: Instructions for Sending Data to IBM i Support

• PM Agent

  NOTE: 
  Effective 30 June 2020, IBM® had withdrawn Performance Management for Power® Systems (PM) services from marketing.
  Support for this product was withdrawn 30 September 2020.
  Additional information and steps to stop sending PM Agent data are included in the Disabling PM Agent document.

  PMAGENT/PM400 free reports, paid reports and website access withdrawn 30 September 2020: https://www.ibm.com/downloads/cas/US-ENUS920-133-CA/name/US-ENUS920-133-CA.PDF

  Withdrawal and service discontinuance: IBM Performance Management for Power Systems (PM) is withdrawn as a stand-alone offering; optional Performance Management reporting is removed from other premium support service offerings.

  Here are the tools available that could be an alternative for PMAgent:

  1. Performance Tools
  2. Performance Data Collectors
  3. Performance on the web - Performance tools for IBM i
  4. Resources
  5. Graph History is an option at 7.3 in Access Client Solutions.  ACS does not support Graph History at 7.1 & 7.2. Additional information can be found at Article: Graph History in IBM Navigator for i Overview
  6. iDoctor Collection Services Investigator - Historical Summaries option
      

• Useful troubleshooting documents

  • Ordering PTFs Using the SNDPTFORD Command
  • Using SNDPTFORD for Image Orders
  • Automatic ordering of PTF groups (QORDPTFGRP job)
  • SNDPTFORD or SNDSRVRQS failing with CPF8C37
  • Replacing Backlevel QASX* Files from Base Operating System Media for Electronic Service Agent Errors
  • Software related problem log entries will not send to IBM
  • Message CPF8C09 - *IBMSRV Not Defined as a Service Provider
  • Message CPIEF3E - Service Agent has detected excessive reportable problems
  • CHGSRVAGTA Fails with messages CPF1015 or CPFEF82
  • ESA: Operations Fails with CPIEF13 - Unable to Use File QAS9AUDLOG
  • ESA: Message CPF2408 - Not authorized to Message Queue QSRV
  • ESA: No Computer System Was Found and the OS is Not Allowed to be Standalone
  • Manage update access keys with IBM Electronic Service Agent (ESA) on IBM i
  • CIMOM server certificate QUMEPRVAGT job not ending