A fix is available
APAR status
Closed as program error.
Error description
For IBM z/OS Explorer 3.3 when a user tries to establish a secure connection with a host system via Remote Systems Explorer, they fail to connect with the following message from the RSE Daemon: RSEG 1056: Cannot connect to server. Unexpected exception. FEKLOGS will show: 68 2023.01.04 12:36:30:977 EST PID:00000232 THREAD:2720880000000036 TCB:005B09B0 USER:GYOUNG DEBUG: class org.eclipse.dstore.core.server.ConnectionEstablisher: SSL/TLS Enabled Protocols: 69 2023.01.04 12:36:30:977 EST PID:00000232 THREAD:2720880000000036 TCB:005B09B0 USER:GYOUNG DEBUG: class org.eclipse.dstore.core.server.ConnectionEstablisher: SSL/TLS Supported Protocols: 70 2023.01.04 12:36:30:977 EST PID:00000232 THREAD:2720880000000036 TCB:005B09B0 USER:GYOUNG DEBUG: class org.eclipse.dstore.core.server.ConnectionEstablisher: TLSv1 71 2023.01.04 12:36:30:977 EST PID:00000232 THREAD:2720880000000036 TCB:005B09B0 USER:GYOUNG DEBUG: class org.eclipse.dstore.core.server.ConnectionEstablisher: TLSv1.1 72 2023.01.04 12:36:30:977 EST PID:00000232 THREAD:2720880000000036 TCB:005B09B0 USER:GYOUNG DEBUG: class org.eclipse.dstore.core.server.ConnectionEstablisher: TLSv1.2 73 2023.01.04 12:36:31:557 EST PID:00000232 THREAD:2720880000000036 TCB:005B09B0 USER:GYOUNG ERROR: class org.eclipse.dstore.core.server.ConnectionEstablisher: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: No appropriate protocol, may be no appropriate cipher suite specified or protocols are deactivated Workaround: 1. Add the following to the end of rse.env: _RSE_JAVAOPTS="$_RSE_JAVAOPTS -Dcom.ibm.jsse2.overrideDefaultTLS=true" 2. Restart the RSE Daemon started task 3. Try to connect to the host system
Local fix
NA
Problem summary
**************************************************************** * USERS AFFECTED: 1. Servers having single logon policy with * * a race condition in concurrent connection * * attempts from the same userID. * * 2. Server running early version of java 11 * * 3. All SSL connections to a server running * * on Java 8 release older than SR6 FP25 * * 4. The server got affected. * * 5. fekfdivp tool * * 6. The server got affected. * * 7. All users running Daemon Operator * * command IVP DAEMON * * 8. All user connection to system having * * tape-type device dataset. * * 9. Users Affected: all RSE connections * * that delay the mvs files systems * * connections in more than passticket timeout. * * 10. all RSEAPI users that issues an mvs * * commands with a delay ~8-10 min after * * starting a connection with a non-mvs * * command.. * * 11. System that wants to use AT-TLS secure * * connection instead of RSE server secure * * connection management. * * The support with client-cert * * authentication mode is introduced in this * * release. * * 12. All users that does not have $SHELL * * profile defined. * * 13. All RSE users using TSO interactive * * gateway and keepAlive option off, invoking * * TSO interactive command. * **************************************************************** * PROBLEM DESCRIPTION: 1. The mvsminer is designed to * * shutdown the logger as it is the main * * user of the logger. * * In a mid of connection startup, this * * miner may not be loaded up to be * * responsible for closing up the logger * * when the connection was shutdown * * midway. * * 2. The pipe count of the ThreadPool * * keeps increasing when running under * * the early version of Java 11. * * When the pipe count (account for the * * total fd cound of the process) * * reaches the process limit, the * * operation would fail. * * The issue does not exist in Java 8. * * Its Process pipe is automatically * * closed when the Process is done. * * * * Another issue that with a heavy load * * ThreadPool especially during user * * logging off, concurrent access issue * * in cleaning up the existing command * * miner threads could stop the cleanup * * abruptly and may cause left over * * resource. * * 3. The Daemon's connection GSK System * * SSL handshake works fine with the * * common protocol TLS v1.2, but the * * Server's Java SSL connection is not * * configured to enable TLS v1.2, * * causing the handshake failed. * * 4. Minimum supported ZOS level for * * Java 1.8 is 2.2 * * 5. New printf call was flagged as * * having possibility of buffer overflow. * * 6. Debugging shows holder of * * stdout/err (of ThreadPools) kept by * * Daemon keeps growing with duplicates. * * Daemon's message listener usually * * would print out garbage when Daemon * * stops. * * 7. The gsk ssl client socket created * * by the tools is required to import * * the trusted server certificate in * * the connection with Daemon through * * AT-TLS layer in the test. Daemon * * non-SSL mode setting prevent the tool * * from configuring the client ssl * * socket properly, causing the failure * * of the test. * * 8. Tape-device type currently is * * logged at info level and as active. * * The active status logging should be * * corrected and at debug level only. * * 9. After the passticket life span is * * expired, during the loading of the * * mvsminer, for the mvs files system * * connection operation, the lock * * manager would need a newly generated * * passticket to start up. The * * generation of the passticket is also * * required to be done under the * * ThreadPool/Daemon user id. * * 10. After the passticket life span * * timeout is expired, during mvsminer * * loading time, for the mvs files * * system initialization and connection * * operation, a request to generate a * * new passticket will be issued and it * * may fail as the user thread may lack * * of the permission to generate the * * pass ticket. * * 11. ATTLS is preferred mechanism for * * securing connections. IBM Developer * * for z/OS, including Enterprise * * Edition should implement ATTLS support. * * The support with client-cert * * authentication mode is introduced in * * this release. * * 12. Command miner when invoked * * without a $SHELL defined will default * * to "sh". * * 13. With TSO interactive gateway, * * when issuing an TSO interactive * * command and entering input, users may * * experience a failure with TSO gateway * * RC 20. * **************************************************************** 1. Left over thread and file descriptor are resource leak to the ThreadPool shared resource for on going and future connections. 2. There is a change in Process instance creation and termination in Java 11. The stdout-in-err pipes of a created Process must be closed explicitly or through a stream wrapper; otherwise, it will cause a leak. Destroying the process also helps. RSED ThreadPool leaks Process pipes through the usage of Java API ProcessBuilder and Runtime exec API to create Process used in its operation. 3. Per IBM Java SDK documentation Matching the behavior of SSLContext.getInstance("TLS") to Oracle - IBM Documentation , the SSLContext of the server is created with "TLS" keyword, which enables TLS 1.0 to TLS 1.2 from Java 8 SR6 PF25; for older version only TLS 1.0 is enabled. 4. Native code for JMON, ZOS, and MVS servers should be compiled with OS target ZOSV2R2 5. printf was known to have a buffer overflow with %s argument. 6. Daemon has the holders for stdout/err fds of ThreadPools to collect their message for logging. It does not reset each round it scans the ThreadPools and keep accumulating duplicates fds. Daemon's message listener process terminates abruptly when exiting causing Daemon end printing out garbage when stops. 7. The client socket should be configured with gsk ssl and attached with the trusted server certificate when Daemon is on AT-TLS mode. 8. Minimize the tape-device info as debug to avoid too much logging for system with high number of tape-device dataset. 9. Lock manager should be started up under user security profile properly with a valid passticket. Only server id is required to have the permission to generate passticket. A user thread may fail to generate a passticket and could fail to load and set up the mvsminer properly in the described scenario. 10. Only server id is required to have the permission to generate passticket. A user thread may fail to generate a passticket and could fail to load and set up the mvsminer properly in the described scenario. 11. AT-TLS support with user/password authenticaiton was introduced in earlier version. The support with client-cert authentication mode is introduced in this release. 12. "/bin/sh" is the proper value for the shell setting. 13. When keepAlive option off, there might be a out of sync to keep the invoking TSO interactive command and the input entering step in the same TSO session, leading to the RC 20 error.
Problem conclusion
1. To fix the problem, in the final step shutting down a client connection, have the DataStore to close the logger again to make sure the logger was cleaned up. Double closing of the same logger handle causes no harm. 2. - Due to the change in Java behavior around the Process pipe management in Java 11, RSED leaks Process pipe during its lifetime causing failure in its operation. Issue has been reported to Java team and awaiting for resolution in Java JDK. A work-around as to explicitly close a Process' pipe when done was introduced. - The common holder of a connection's command miner threads should be synchronized during cleanup or adding operation. 3. The listing of enabled protocols under the Protocols standard name is not sufficient, must refer to specific documentation on SSLContext.getInstance on matching the behavior to Oracle for proper usage. Have the Server's context created with the keyword "SSL" to enable TLSv1.0 to TLSv1.2 support. 4. Update build script with compile OS target set to ZOSV2R2 5. The scan result is actually a noise as printf was used with argument constructed internally having bounded message. But introduce the alternative of sprintf and puts anyway, avoiding marking it as noise. 6. Reset the std fd holder in each round of scan. Have the messaging process sending an exit ack back to Daemon for its message listener to display properly. 7. Have the client socket to detect AT-TLS mode of Daemon. Specify the key ring in daemon_keydb_file parameter for the tool to access and uses in ssl configuration of its client socket. 8. Tape-device type active status is corrected and logged only at debug level 9. Have the lock manager startup with valid passticket. Have passticket generation call used by the mvs Files System initialization and connection operation to run in a newly generated thread to inherit the process server id to be able to generate the passticket. Connection should be done with the newly generated ticket for the mvsminer's lock manager to work with proper security profile as the user. 10. Have passticket generation call used by the mvs Files System initialization and connection operation to run in a newly generated thread to inherit the process server id to be able to generate the passticket. Connection should be done with the newly generated ticket for the mvsminer's lock manager to work with proper security profile as the user. 11. Have the client send the client certificate in data mode as with the existing RSE ssl workflow. RSED ssl for the key ring is required for the client-cert validation as with the current ssl configuration. 12. When it is not defined, have $SHELL default to the proper value "/bin/sh" instead of "sh". 13. To adjust the timing of the keepAliveThread's pinging execution for an interactive command in waiting input mode. Recalculate the wait (sleep) time for the pinging so that it could happen on time to keep the session valid w.r.t RSE CEA TSO processor and gateway client session timeout. It is to synchronize and keep alive the session to be used for both command issuing and input entering step when the RSE host is not running with the CEATSO keep alive option.
Temporary fix
Comments
Keyword: ZOS0301T/K
APAR Information
APAR number
PH51845
Reported component name
EXP FOR Z/OS HO
Reported component ID
5655EXP23
Reported release
330
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2023-01-11
Closed date
2023-03-10
Last modified date
2023-05-02
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UI90920
Modules/Macros
FEJENF70 FEJJCNFG FEJJJCL FEJJMON FEJTSO FEK1SMPE FEK2RCVE FEK3ALOC FEK4ZFS FEK5MKD FEK6DDEF FEK7APLY FEK8ACPT FEK@CERR FEK@CONE FEK@CONF FEK@CUST FEK@DEB FEK@DESC FEK@FLOW FEK@GEN FEK@GENW FEK@ISPF FEK@IVP FEK@IVPD FEK@IVPW FEK@JCN1 FEK@JCNE FEK@JESJ FEK@MAIN FEK@MIGO FEK@OPTE FEK@OPTG FEK@OPTN FEK@PRIM FEK@RSE1 FEK@RSEO FEK@STRT FEK@TAB1 FEK@TAB2 FEK@TAB3 FEK@WRK1 FEK@WRK2 FEK@WRK3 FEK@WRK4 FEK@WRK5 FEKAPPCC FEKAPPCL FEKAPPCX FEKATTR FEKDSI FEKEESX0 FEKFASIZ FEKFATT1 FEKFBLD FEKFCIPH FEKFCLIE FEKFCMOD FEKFCMPR FEKFCMSG FEKFCOMM FEKFCOPY FEKFCOR6 FEKFCORE FEKFDBG FEKFDBG6 FEKFDBGM FEKFDIR FEKFDIR6 FEKFDIVP FEKFDST0 FEKFDST1 FEKFDST2 FEKFENVF FEKFENVI FEKFENVP FEKFENVR FEKFENVS FEKFEPL FEKFERRF FEKFGDGE FEKFICUL FEKFISPF FEKFIVP0 FEKFIVPA FEKFIVPD FEKFIVPI FEKFIVPJ FEKFIVPT FEKFJESM FEKFJESU FEKFJLIC FEKFJSON FEKFJVM FEKFLATR FEKFLDSI FEKFLDSL FEKFLEOP FEKFLOGS FEKFLPTH FEKFMAI6 FEKFMAIN FEKFMINE FEKFMNTL FEKFNTCE FEKFOMVS FEKFPATT FEKFPLUG FEKFPTC FEKFRIVP FEKFRMSG FEKFRSES FEKFRSRV FEKFSCMD FEKFSEND FEKFSSL FEKFSTUP FEKFT000 FEKFT002 FEKFTIVP FEKFTSO FEKFUTIL FEKFVERS FEKFXITA FEKFXITL FEKFZOS FEKHCONF FEKHCUST FEKHDEB FEKHDESC FEKHFLOW FEKHGEN FEKHISPF FEKHIVP FEKHIVPD FEKHJESJ FEKHMAIN FEKHMIGO FEKHOPTE FEKHOPTN FEKHPRIM FEKHRSE1 FEKHRSEO FEKHSTRT FEKHTAB1 FEKHTAB2 FEKINIT FEKKEYS FEKLOCKA FEKLOGR FEKLOGS FEKM00 FEKM01 FEKM02 FEKMKDIR FEKMOUNT FEKMSGC FEKMSGS FEKRACF FEKRSED FEKSAPF FEKSAPPL FEKSBPX FEKSCLAS FEKSCLOG FEKSCMD FEKSCPYM FEKSCPYU FEKSDSN FEKSENV FEKSETUP FEKSISPF FEKSJCFG FEKSJCMD FEKSJMON FEKSLPA FEKSPROG FEKSPTKT FEKSRSED FEKSSERV FEKSSTC FEKSSU FEKSUSER FEKXCFGE FEKXCFGI FEKXCFGM FEKXCFGT FEKXMAIN FEKXML HUHFCOR6 HUHFCORE
Fix information
Fixed component name
EXP FOR Z/OS HO
Fixed component ID
5655EXP23
Applicable component levels
R330 PSY UI90920
UP23/03/18 P F303
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSBDYH","label":"IBM Explorer for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"330","Line of Business":{"code":"LOB35","label":"Mainframe SW"}}]
Document Information
Modified date:
02 May 2023