A fix is available
APAR status
Closed as program error.
Error description
When CICS is invoking a program or programs enabled to run at the XPCFTCH Global User Exit point, the XMLOCK is acquired Exclusive before getting some information to be put in the DFHUEPAR parameter list to be passed to the GLUE. Getting the XMLOCK Exclusive at this point can cause excessive LMQUEUE wait time if there is lots of contention for the XMLOCK. (One thing that can cause lots of contention for the XMLOCK is lots Starts for new transactions while there are thousands of transactions queued for TCLASS or MXT reasons.) . Here is an example of the Kernel stack of a transaction that is waiting on the XMLOCK while trying to invoke an XPCFTCH GLUE: . TYPE ADDRESS LINK REG OFFSET ERR NAME Bot 2E503700 AE503CB8 0005B8 DFHKETA Dom 2E51F638 AE51F8CE 000296 DFHDSKE Dom 2E559D18 AE55B7B6 001A9E DFHXMTA Dom 2EC1C098 AEC1F06C 002FD4 DFHPGPG Int +0003D4 AEC1C26C 0001D4 INITIAL_LINK Dom 2F115600 AF11AC88 005688 DFHAPLI1 Int +00315E AF115FE2 0009E2 DRIVE_XPCFTCH_EXIT Int +005642 AF11879A 00319A SETUP_GLUE_PARAMETERS Dom 2E562450 AE5637E8 001398 DFHXMIQ Int +00131C AE562630 0001E0 GET_XM_LOCK Dom 2E53AB28 2E53C359 001831 DFHLMLM Int +0006C0 2E53AE16 0002EE LOCK Int +00104C 2E53B21E 0006F6 LOCK_EXCLUSIVE Int +0007B8 2E53BC76 00114E ADD_TO_LOCK_QUEUE Int +001774 2E53B470 000948 SUSPEND Dom 2E512398 AE51432A 001F92 DFHDSSR Int +00192E AE512E88 000AF0 POP_TASK . Not having to get the XMLOCK exclusive in this situation would remove a bottleneck point. That would improve performance in situations such as described above. . Additional Symptom(s) Search Keyword(s): KIXREVDWZ
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All CICS users * **************************************************************** * PROBLEM DESCRIPTION: Poor response times and performance * * degradation due to XMLOCK * * contention. * **************************************************************** A flurry of event processing activity led to CEPD transactions trying to attach many thousands of tasks to process the events. There was a degradation in system performance and throughput as a result of this workload. Many of the user tasks that were attached were suspended in LMQUEUE waits. The lock being waited upon was the XMLOCK. The user tasks needed to get the XMLOCK exclusive when the initial program was being invoked. This is because there was a GLUE that was enabled to run at the XPCFTCH exit point. CICS passes some transaction-related parameters to the exit. It gets the XMLOCK before getting those transaction-related parameters because it issues a DFHXMIQM domain call. In addition, the CEPD transactions need to get the XMLOCK exclusive when they attach a user transaction to process the events. As the TCLASS queue grows, there is some CPU intensive code within CICS transaction attach logic that runs while the XMLOCK is held. This compounded the general performance issue because CICS needs to run the transaction chain each time it goes to attach a transaction. The result of these factors means that getting the XMLOCK is a bottleneck within a CICS system as the number of CEPD transactions reaches several hundred. This causes the user transactions to have lots of Lock Delay time and wait-for-redispatch time.
Problem conclusion
DFHAPLI1 has been changed to use an inlined DFHXMIQI call in place of the DFHXMIQM domain call.
Temporary fix
Comments
APAR Information
APAR number
PH09282
Reported component name
CICS TS Z/OS V5
Reported component ID
5655Y0400
Reported release
000
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2019-03-04
Closed date
2019-05-20
Last modified date
2019-06-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
010PC2 UI63103 UI63104 UI63105
Modules/Macros
DFHAPLI1 DFHAPLX1 DFHAPLX3
Fix information
Fixed component name
CICS TS Z/OS V5
Fixed component ID
5655Y0400
Applicable component levels
R000 PSY UI63103
UP19/05/21 P F905
R100 PSY UI63104
UP19/05/21 P F905
R200 PSY UI63105
UP19/05/21 P F905
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.3","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.3","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
03 June 2019