Troubleshooting ABAP extraction jobs that use Remote Function Call (RFC) data transfer method

Resolving The Problem

For performance tuning these specific steps apply: 3, 11, 12, 14, 15, 16, 17
For generic troubleshooting complete the steps in order or identify the steps which may apply to your specific environment.

Notes:

• The transactions (t-code) specified are completed in the SAP GUI application (SAP Logon)
• These steps are generic for all RFC type data transfer (tRFC, qRFC, bgRFC). For additional steps specific to bgRFC, see technote 2001266

Steps:

1. In the ABAP stage GUI, use the Validate button (Output/General tab) to check the stage settings; make sure all checks are green before proceeding. Any errors reported should be corrected (either connection or SAP transport related). Note the validation uses a connection from the Client tier to SAP while the runtime connection is done from the Engine tier.
2

2. Verify the required Pack prerequisites are installed on both Engine and Client tiers, including the SAP SDK libraries. A link to the prerequisites technote is included in the Release Notes. For example for Pack version 8.1 or 8.2: http://www.ibm.com/support/docview.wss?uid=swg22016343
Missing prerequisites on the Engine tier may produce an unhandled exception reported in the job log.

3. Jobs using the RFC data transfer method and running at the same time must be configured to use a unique RFC destination.

• Make sure the ABAP stage meets the requirements specified in Technote 1314075.
• To manually verify the uniqueness of these values across jobs see Technote 1966231.

Note: We have recently implemented (Sep 2016) an enhancement for automatic detection of duplicate Destinations and Program IDs. The update is available as a patch (for Server and SAP tiers) for maintenance versions 7.1 and 8.0. It's also included in version 8.0.0.1 (Fixpack 1 for 8.0).
Note there are a couple DS core patch prerequisites documented in the APAR entry below.
APAR JR50473: ENHANCE GENERATED ABAP TO DETECT IF ANOTHER PROCESS WITH SAME RFC DESTINATION NAME EXISTS AND ACTIVE AT TIME OF THE QUERY.

4. This type of error in the job log "registration of <destination> from host <hostname> not allowed"
means registration of RFC destination is not allowed from Datastage to the SAP host. Check SAP Profile parameter gw/acl_mode using t-code RZ11. If value is 1 make it 0 (zero) to allow registration. Example of full error:
^{Fatal: RFC server function 'rfc_getline' failed, returned '765:ERROR: RFC error has occured with the following details: %1:Error Details: BAPI: RfcListenAndDispatch Code: RFC_COMMUNICATION_FAILURE Key: RFC_COMMUNICATION_FAILURE Message: LOCATION SAP-Gateway on host SRVBRA153.tods.com / sapgw00
ERROR registration of tp ZRFC_ZRT_TP_CALC_IVA from host srvibm4bids.tods.com not allowed}

5. There's a known issue with the stage tracing in versions 6.5 - 8.0, where it may cause a job abort (or other unexpected symptoms) for certain data sets. Make sure to test with stage tracing off as well (unset environment variable DSSAP_STAGE_RUNTIME_TRACE or set it to NONE).

6. Install the latest Service Pack for your SAP system. Similar issues in the past were fixed by applying SAP Note 977283.
Also, verify the SAP system has not run out of memory - use OS specific tools. Check the SAP system logs as well.

7. If this is the first time that the job or connection is set up or if you have multiple jobs in this environment, troubleshoot connection errors by completing the steps in Technote 1393028.

8. If the job fails with a timeout message, set the background job timeout limit regardless of whether you use a background job or not.

To configure a timeout limit, complete one of the following steps:

• Change the stage property BG_JOB_TIMEOUT. For more information, see Technote 1576635.
• Set the environment variable DSR3_BG_JOB_TIMEOUT. For more information, see APAR JR38704.

The default timeout limit is 600 seconds. When you specify the timeout limit, consider the system performance and the extraction time. For very heavy loads, increase the timeout interval to several hours if required. The setting affects non-background operations as well.

9. Verify connectivity between the InfoSphere Information Server engine tier and the SAP system. To verify connectivity, use transaction SM59:

• 1. Run the ABAP extraction job.
  2. While the job is running, test the connection, including the Unicode test.

Note: When you test connectivity, the job must be running because the dynamically created RFC destinations are deleted after the job completes.

10. If you use background jobs, use transaction SM37 to check the status. The name of the background job is the name of the ABAP program. You can delete any jobs that are no longer needed.

11. Increase the value for the Dialog Workprocesses parameter in the SAP system. Increase the value set for the parameter "rdisp/wp_no_dia" parameter using one of the transactions:

• For a temporary change, active until next instance restart: In transaction RZ11 (Maintain Profile Parameters), enter the parameter name and then click Change (you may have to enter the maintenance mode first by clicking Goto > Maintenance Mode).
• For a permanent change, active after the next instance restart: In transaction RZ10 (Edit Profiles) select the instance profile, check extended maintenance and click Change. Edit the parameter in the list, then save and activate the profile.

Note: Before making any changes to the SAP system parameters, discuss the changes with the local SAP Basis administrator. See also other relevant SAP system parameters listed below.

12. If you use static RFC destinations:

• Use transaction SM59 to verify the destination for the transported data. Verify that the destination is configured correctly and that Unicode is specified in the special options.
• If you cannot increase the number of Dialog Workprocesses, check the maximum number of connections for the RFC destination. If a high value is used, specify a lower value by completing the following steps:
  1. In transaction SMQS, select the destination.
  2. Press F6 and change the value that is specified for the Max Connections field.

Specify a number that is less than or equal to the number nodes that are available in InfoSphere Information Server. For more information, see SAP Note 1403974.

13. Verify the CPI-C connection timeout value for the destination is set to be longer than the job execution time:

• 1. In transaction SMQS, select the destination.
  2. Press F6 and verify the value that is specified for the Max Runtime field.

To increase the default connection timeout value (SAP Gateway parameter gw/reg_timeout) complete one of the following steps:

• for a temporary change, active until next instance restart: In transaction RZ11 (Maintain Profile Parameters), enter the parameter name and then click Change (you may have to enter the maintenance mode first by clicking Goto > Maintenance Mode).
• for a permanent change, active after the next instance restart: In transaction SMGW (Gateway Monitor) click Goto > Parameter > Change.

Note: A related parameter that should be considered for increase is gw/cpic_timeout

14. The environment variable CPIC_MAX_CONV can be defined at OS level (both on the SAP Server and the DS Server). It controls the maximum number of CPIC conversations that can be initiated (default is 100). Should be increased if getting an error "max no of conversations exceeded" or to tune performance.

15. If you use tRFC, use transaction SM58 (Transactional RFC) to delete outstanding failed transactions. In addition, to automate the cleanup:

• In IBM InfoSphere Information Server Pack for SAP Applications, Version 7.0 and later, to automate deletion of failed transactions, complete the following steps:
  1. Open the ABAP stage properties.
  2. Click Output > Data Transfer Method, and then select Delete leftover LUWs.
  3. Recompile the job.
• In version 6.5 implement the Z_DELLUW cleanup program. For more information, see APAR JR38586.

Note: If the job still hangs after deleting the transactions in SM58 and automated cleanup, perform manual tRFC cleanup. In transaction SE38 run the report RSARFCER (Delete tRFC); enter a Date range, Destination (can use *) and User Name. From the "Status" check boxes select everything that applies, then click Execute (or press F8).

16. If you use tRFC, use transaction SM58 (Transactional RFC) to inspect the transactional status while the job is running. If the transaction status is "Recorded", the SAP System is supposed to execute that transaction via the scheduler. If the transaction remains "Recorded" try to execute it manually in the SAP GUI. If the job now completes fine, it means the issue is with the SAP scheduler - the issue needs to be resolved by the local SAP Basis administrator.

17. If you use parallel jobs, verify the combinability mode for the ABAP stage indicates the stage runs in its own process. Also, make sure more than one parallel node is available to the stage.
First, configure the DataStage parallel environment to make available at least 2 nodes to the job (see the DataStage documentation for details). The number of available nodes is displayed by the ABAP stage in the job log, e.g. Message: ABAP1,0: Info: Job1.ABAP1: Number of parallel nodes: 4.
Then, configure the stage:

1. Open the ABAP stage properties
2. Click Stage > Advanced
3. Make sure Execution Mode = Parallel and Combinability mode = Don't combine
4. Recompile the job if any changes.

Note: an alternative method for disabling combinability only is setting the APT_DISABLE_COMBINATION environment variable to 1. Although the environment variable is set just for the specific job, it affects all stages in the job. If you have the permission to change the job, then it is recommended to use the stage level setting.

18. Use transaction SE38 to run the ABAP extraction program manually:

• 1. Open the job in the InfoSphere DataStage Designer client
  2. Open the ABAP stage
  3. Click Output > ABAP Program, and then change the run method to "SAP administrator runs the program manually"
  4. Regenerate the program
  5. Re-compile and re-run the job
  6. Run the program manually from transaction SE38.
  7. Check the program status in transaction sm50.
• To check if the ABAP extraction stage is able now to start the program automatically, complete the following steps:
  • • Open the ABAP stage.
    • Click Output > ABAP Program, and then change the run method to "DataStage job runs the program"
    • Regenerate the program
    • Re-compile and re-run the job.

19. If the ABAP job continues to hang:

• 1. Open the ABAP stage properties
  2. Change the name of the ABAP program
  3. Change the value for the RFC destination and the program ID to new unique values
  4. Regenerate the ABAP program.

20. The latest maintenance code may contain relevant fixes. Users can request a patch by opening a PMR. The patches are cumulative, so requesting the latest maintenance code will include all available fixes. If the issues persist after installing the Client and Engine tiers patches and regenerating the programs, the PMR can be used to have the issue investigated further.

Other useful SAP transactions (t-code):
SM37 - Job Selection - if using background programs can check the job status (background job name is the ABAP program name)
ST11 - Error Log Files; displays all traces on the kernel: workprocess (dev_wxxx), RFC calls (dev_rfcxxx), gateway (dev_rdxxx)
ST22 - ABAP Runtime Error
SM50 - Process Overview, can see the status of the running ABAP program
SM21 - System Log
SM66 - Global Work Process Overview

Other SAP system parameters affecting tRFC and qRFC transactions:
RDISP/ASYNC_DIALOG_TIMEOUT - timeout for asynchronous dialog RFCs
RDISP/RFC_POOL_TIMEOUT - Timeout for pooled RFC connections
RFC/REDUCE_TRACELEVEL - reduces the RFC-information which is written into the WP and RFC-Traces
DBS/MSS/ADD_PROCS - maximum number of additional dirty read connections
GW/CONN_PENDING - timeout for pending connections
GW/GW_DISCONNECT - Timeout to cancel inactive gateway connections