Setting the preferred connector to RMI on admin service caused an IO connection refused in following code: com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl createSSLSocket

Answer

The Nodeagent (NA) log shows that it is attempting to make a client connection to another JVM (dmgr? appserver?) and that connection fails because the server side is not listening on that particular port.

ORBRas E com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl createSSLSocket
Non-deferrable Alarm : 1 JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred. Reason: Connection refused: connect java.net.ConnectException: Connection refused: connect
...
at com.ibm.ws.management.connector.rmi._RMINotificationListener_Stub.
handleNotifications
at com.ibm.ws.management.connector.rmi.RMINotificationListenerAdapter.
handleNotifications
at com.ibm.ws.management.event.PushRemoteSender.pushNotifications
at com.ibm.ws.management.event.AbstractPushRemoteSender.alarm
at com.ibm.ejs.util.am._Alarm.run

Connection refused usually means either
1. The server side is being recycled and not yet listening on the port in question. 2. The server side is simply not up and running
3. The client (Nodeagent (NA) in this case) has a stale IOR which has old host/port info (because the server has dynamic ports and has been recycled).

NA_stopped_first logs (Dmgr log):

Nodeagent (NA) sends Dmgr a close connection message since it is now shutting down. Shortly thereafter, the NodeAgentPing thread receives the "Connection refused" exec when trying to connect to 9118 (SSL 9211).

ORBRas E com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl createSSLSocket
NodeAgentPing JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred. Reason: Connection refused: connect java.net.ConnectException: Connection refused: connect

This is expected behavior as the Dmgr will continue to do isAlive calls to the NA and when the NA finally goes down, we receive the "connection refused" exec until the NA is back up and running.

Dmgr_stopped_first logs (NA trace.log):

Up until the Dmgr sends the NA a Close Connection message, communication is fine and NA is sending Dmgr "handleNotifications" reqs throughout. Dmgr sends NA close connection message (Dmgr shutting down). NA tries to make new connection to Dmgr for a new handleNotifications() call and gets "connection refused" because Dmgr is down. This is expected.