APAR status
Closed as program error.
Error description
After upgrade to IBM MQ v8.0.0.6 (or later) Managed File Transfer, agents running under heavy load abend. When the abend occurs, the following messages are written to the agent's event log (output0.log) 000002de StdErr E java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 72 000002de StdErr E at com.ibm.crypto.fips.provider.af.a(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.af.decrypt(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.W.decrypt(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.P.b(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.at.a(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.at.a(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.at.b(Unknown Source) 000002de StdErr E at com.ibm.crypto.fips.provider.at.engineDoFinal(Unknown Source) 000002de StdErr E at javax.crypto.Cipher.doFinal(Unknown Source) 000002de StdErr E at com.ibm.wmqfte.utils.CredentialsFileEncoder.decodeUnwrapped(Cred entialsFileEncoder.java:161) 000002de StdErr E at com.ibm.wmqfte.utils.CredentialsFileHelper.getFilePassword(Crede ntialsFileHelper.java:388) 000002de StdErr E at com.ibm.wmqfte.transfer.TransferConnectionDetailsFactory.getConn ectionDetails(TransferConnectionDetailsFactory.java:87) 000002de StdErr E at com.ibm.wmqfte.transfer.impl.TransferReceiverRunnable.run(Transf erReceiverRunnable.java:398) 000002de StdErr E at java.lang.Thread.run(Thread.java:809) 000002de StdErr E at com.ibm.wmqfte.thread.FTEThread.run(FTEThread.java:70) 000002de ABENDImpl E BFGUT0003E: An internal error has occurred. An unexpected condition has occurred that has resulted in the abnormal end of this process. Failure data was captured in file "ABEND.FTE.YYYYMMDDHHMMSSxxx.xxxxxxx.log". The ABEND.FTE.YYYYMMDDHHMMSSxxx.xxxxxxx.log file shows: Class: com.ibm.wmqfte.thread.FTEThread ethod: uncaughtException Probe: ABEND_001 Cause: java.lang.NullPointerException java.lang.NullPointerException at com.ibm.crypto.fips.provider.af.a(Unknown Source) The NullPointerException has the same stack as shown in the event log (output0.log) for the agent. The errors do not happen when IBM MQ Managed File Transfer agent trace is enabled. Additional Symptom(s) Search Keyword(s): MFT transfer java.lang.NullPointerException java.lang.ArrayIndexOutOfBoundsException com.ibm.crypto.fips.provider.af.a
Local fix
restart agent
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - IBM MQ V8.0.0.6 (and later) Managed File Transfer. - IBM MQ V9 Managed File Transfer. who have agents that have been configured to use an IBM MQ Managed File Transfer credentials file, where the sensitive data in the credentials file has been encrypted using the fteObfuscate command. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: If the queue manager for an IBM MQ Managed File Transfer (MFT) agent has been configured with security enabled, and requires a user identifier and password to be flowed when the agent connects, then the user identifier and password can be stored in an IBM MQ MFT credentials file (MQMFTCredentials.xml). By default, agents will try to load the IBM MQ MFT credentials file from the home directory of the user that the agent is running as. To specify a different location, the property agentQMgrAuthenticationCredentialsFile needs to be set in the agent.properties file for the agent. The password stored in the IBM MQ MFT credentials file can be encrypted using the fteObfuscate command. All of the threads within an agent use the same internal object whenever they need to decrypt a password contained within the IBM MQ MFT credentials file. This internal object makes use of some functionality provided by the underlying Java Runtime Environment (JRE) in order to perform the decryption. Now, if an agent was performing lots of managed transfers simultaneously, then it was possible for multiple threads within an agent to be decrypting passwords at the same time. When using IBM MQ V8.0.0.5 (or earlier) MFT, the functionality provided by the Java Runtime Environment allowed this behaviour. However, the Java Runtime Environments shipped with IBM MQ V8.0.0.6 (and later), and IBM MQ V9, would throw ArrayIndexOutOfBoundsExceptions or NullPointerExceptions if two or more threads were performing decryption operations simultaneously. Because the threads within the agent were not expecting these exceptions to be thrown, they generated an abend file (called "ABEND.FTE.<date and timestamp><number>.<number>.log") and then shut the agent down. Some examples of the abend files that were generated are shown below: ABEND File 1: ----------------- Class: com.ibm.wmqfte.thread.FTEThread Method: uncaughtException Probe: ABEND_001 Cause: java.lang.NullPointerException java.lang.NullPointerException at com.ibm.crypto.fips.provider.af.a(Unknown Source) at com.ibm.crypto.fips.provider.af.decrypt(Unknown Source) at com.ibm.crypto.fips.provider.W.decrypt(Unknown Source) at com.ibm.crypto.fips.provider.P.b(Unknown Source) at com.ibm.crypto.fips.provider.at.a(Unknown Source) at com.ibm.crypto.fips.provider.at.a(Unknown Source) at com.ibm.crypto.fips.provider.at.b(Unknown Source) at com.ibm.crypto.fips.provider.at.engineDoFinal(Unknown Source) at javax.crypto.Cipher.doFinal(Unknown Source) at com.ibm.wmqfte.utils.CredentialsFileEncoder.decodeUnwrapped(Cred entialsFileEncoder.java:161) at com.ibm.wmqfte.utils.CredentialsFileHelper.getFilePassword(Crede ntialsFileHelper.java:388) .............. ABEND File 2: ----------------- Class: com.ibm.wmqfte.thread.FTEThread Method: uncaughtException Probe: ABEND_001 Cause: java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException 	at com.ibm.crypto.fips.provider.at.a(Unknown Source) 	at com.ibm.crypto.fips.provider.at.engineUpdate(Unknown Source) 	at com.ibm.crypto.fips.provider.at.b(Unknown Source) 	at com.ibm.crypto.fips.provider.at.engineDoFinal(Unknown Source) 	at javax.crypto.Cipher.doFinal(Unknown Source) 	at com.ibm.wmqfte.utils.CredentialsFileEncoder.decodeUnwrapped(Cred entialsFileEncoder.java:161) 	at com.ibm.wmqfte.utils.CredentialsFileHelper.getFilePassword(Crede ntialsFileHelper.java:388) ..............
Problem conclusion
In order to resolve this issue, locking has been added to the internal object used by IBM MQ MFT agents to ensure that only one thread is decrypting a password stored within an IBM MQ MFT credentials file at any one time. This prevents the ArrayIndexOutOfBoundsException or NullPointerException being thrown from the Java Runtime Environment, and means that the agent will no longer abend. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v8.0 8.0.0.11 v9.0 LTS 9.0.0.5 v9.x CD 9.1.0.0 The latest available MQ maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT25264
Reported component name
IBM MQ MFT V8.0
Reported component ID
5724H7252
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-06-04
Closed date
2018-06-08
Last modified date
2020-11-16
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ MFT V8.0
Fixed component ID
5724H7252
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0"}]
Document Information
Modified date:
18 November 2020