Troubleshooting
Problem
This document contains troubleshooting information for the Secure Sockets Layer (SSL) component in WebSphere® Application Server traditional. This can help address common issues with this component before calling IBM support and save you time.
Resolving The Problem
Runtime: Overview
This topic contains error messages and common issues that can require an SSL trace to determine the root cause of the problem. The instructions to obtain an SSL trace are in the 'Collecting data manually' section of the Collect data tab. If a trace string different than what is on the Collect data is required for a specific problem, that trace string is be noted in the steps to diagnose the problem.
I applied WebSphere fix pack 8.5.5.6 to disable SSLv3 from WebSphere Application Server, but SSLv3 is still enabled. How can I fix this issue?
The default installation of WebSphere Application Server 8.5 uses JDK 1.6. One thing to be on the lookout is that if you're using JDK 1.7, the JDK is not automatically updated when you install the WebSphere fix pack. It is possible that you have updated your WebSphere Application Server fix pack, but you're still running an old version of JDK 1.7, which does not contain the fix to disable SSLv3.
If you're using JDK 1.7, update it using the Installation Manager to an SR level that is at, or greater than one of those listed on Security Bulletin: Vulnerability in SSLv3 affects IBM WebSphere Application Server (CVE-2014-3566).
Seeing CWPKI0045E message in the logs
I'm looking at the logs and I see the following message CWPKI0045E, which has confusing information in it:CWPKI0045E: SSL HANDSHAKE FAILURE: A signer with Subject DN "CN=aaaa, OU=bbbb, O=cccc, L=dddd, ST=ee, C=ff" was sent from a target host. The signer may need to be added to local trust store "unknown:0" located in SSL configuration alias "/cts/wp/etc/DummyServerTrustFile.jks" loaded from SSL configuration file "NodeDefaultSSLSettings". The extended error message from the SSL handshake exception is: "security.xml" |
Discussion on dwAnswers.
There is an error in the displaying of CWPKI0045E that is fixed on APAR PI60398 in fix packs 7.0.0.43, 8.0.0.13 and 8.5.5.10.
You should see something similar to:
CWPKI0045E: SSL HANDSHAKE FAILURE: A signer with Subject DN "CN=aaaa, OU=bbbb, O=cccc, L=dddd, ST=ee, C=ff" was sent from a target host. The signer may need to be added to local trust store "/cts/wp/etc/DummyServerTrustFile.jks" located in SSL configuration alias "NodeDefaultSSLSettings" loaded from SSL configuration file "security.xml". The extended error message from the SSL handshake exception is: |
How do I fix the SSL exception CWPKI0022E "Extended key usage does not permit use for TLS client authentication"
In my SystemOut.log I can see the following SSL exception:CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN "CN=abc, OU=IT, O=ibm , C=US" was sent from target host:port "unknown:0". The signer may need to be added to local trust store "/opt/IBM/WebSphere/AppServer/profiles/Dmgr/config/cells/DmgrCell/trust.p12" located in SSL configuration alias "XDADefaultSSLSettings" loaded from SSL configuration file "security.xml". The extended error message from the SSL handshake exception is: "Extended key usage does not permit use for TLS client authentication". |
Discussion on dwAnswers.
The problem here is with CA certificate itself. You need to request a new CA to fix it.
|
Problem determination:
Examine your CA certificate with iKeyman or similar tool. Check for the Criticality and ExtKeyUsage settings. Given this error, you probably have Criticality=false, ExtKeyUsage [1.3.6.1.5.5.7.3.1]. In this case, ExtKeyUsage indicates it is for server authentication, which requires that the Criticality flag be true, but it is false, which is not valid.
Solution: Certificates should be regenerated setting the Extended Key Usage Criticality attribute to true.
Error in WebSphere Application Server logs JSSL0130E java.net.SocketTimeoutException: Read timed out
When I start the Application server, I see the following error in Application server systemout.log and node agent systemout.log:ORBRas E com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl createSSLSocket ORB.thread.pool : 0 JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred. Reason: Read timed out Remote Host: 1.2.3.4 Remote Port: 9202 java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:140) at ava.net.ManagedSocketInputStreamHighPerformanceNew.read(ManagedSocketInputStreamHighPerformanceNew.java:274) at com.ibm.jsse2.b.a(b.java:245) at com.ibm.jsse2.b.a(b.java:229) at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:670) ... at com.ibm.ws.orb.services.lsd._ORB_ServerStub.ping(_ORB_ServerStub.java:34) at com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl.reactivateServers(LocationServiceImpl.java:885) at com.ibm.ws.orbimpl.services.lsd.LocationServiceImpl.getNextTarget(LocationServiceImpl.java:430) at com.ibm.ws.orbimpl.services.lsd.LocationServiceDaemon.getDirectIOR(LocationServiceDaemon.java:203) |
Discussion on dwAnswers.
Resolving the problem:
It is often sufficient to increase timeouts for RMI and SSL. Do the following:
com.ibm.ws.orb.transport.SSLHandshakeTimeout=60000 (1 minute) |
com.ibm.CORBA.requestTimeout=180 |
- From the client scripts interacting with WebSphere over RMI, set the property
- In sas.client.props (or its equivalent in use from client scripting, set the property
- In the admin console, do the following:
com.ibm.ws.orb.transport.SSLHandshakeTimeout=60000
com.ibm.CORBA.requestTimeout=180- Navigate to System Administration -> Deployment Manager -> ORB Service -> Custom Properties
- Add the following properties:
- Do the same for the node agent and for servers, specifically if wsadmin scripts or commands are failing with RMI/Corba issues.
- If the issue involves wsadmin commands that involve work on AppServers, it might also be necessary down to that level. Especially on large nodes with many servers.
Performance Issues with InetAddress Resolutions
By default, InetAddress resolutions requiring the network will be carried out over IPv6 if the operating system supports both IPv4 and IPv6. However, if the name service does not support IPv6, then a performance issue might be perceived as the doomed IPv6 query must first timeout before a successful IPv4 query can be made.To reverse the default behavior and use IPv4 over IPv6, add the following Generic JVM argument to you your dmgr, node agent and all application servers:
-Djava.net.preferIPv4Stack=true |
If you still experience the problem, then make sure there is no other network, operating system, or firewall issues.
Error while converting certificates to 2048 bits and Sha256 Algorithm
I am working on converting certificates to 2048 bits and Sha256 Algorithm. I used this link as a reference: How do I change the WebSphere default certificate to 2048 bits and Sha256 Algorithm in 7.0.0.23 and above?When I perform this step, I get an error:
wsadmin>$AdminTask convertCertForSecurityStandard {-fipsLevel FIPS140-2 -signatureAlgorithm SHA256withRSA -keySize 2048} WASX7015E: Exception running command: "$AdminTask convertCertForSecurityStandard {-fipsLevel FIPS140-2 -signatureAlgorithm SHA256withRSA -keySize 2048}"; exception information: com.ibm.websphere.management.cmdframework.CommandException com.ibm.websphere.crypto.KeyException: com.ibm.websphere.crypto.KeyException: A signer certificate with alias: CN=localhost, OU=Root Certificate, OU=localhostCell01, OU=localhostCellManager01, O=IBM, C=US already exists but it contains a different public key |
To resolve it, I tried using this document: Certificates will need to be converted to use SHA256withRSA in WebSphere Application Server.
I get a new error:
wsadmin>$AdminTask convertCertForSecurityStandard {-fipsLevel FIPS140-2 -signatureAlgorithm SHA256withRSA -keySize 2048} WASX7015E: Exception running command: "$AdminTask convertCertForSecurityStandard {-fipsLevel FIPS140-2 -signatureAlgorithm SHA256withRSA -keySize 2048}"; exception information: com.ibm.websphere.management.cmdframework.CommandException com.ibm.security.certclient.base.PkRejectionException: com.ibm.security.certclient.base.PkRejectionException: 3008-706 Certificate could not be found in the keystore. Keystore alias provided may be incorrect. (wraps: java.security.cert.CertificateParsingException: X.509 Certificate is incomplete: SubjectAlternativeName extension must be marked critical when subject field is empty) |
Discussion on dwAnswers.
The convert option can only change the default certificate in keystores. If you have any other certificate, such as a self-signed or CA certificate, then it will not convert.
You might have a non-default certificate in one of your keystores that is causing the issue. The convertCertForSecurityStandard admin task will check all keystores and truststores referred in security.xml. If any of the keystores or truststores has an issue, then the command will fail.
Problem determination:
Check all the keystores and truststores referenced in security.xml to see if they have any certificates which are not signed by the default WebSphere root certificate for the cell, for example, a self-signed certificate or a CA-signed certificate.
To find the personal certificates in security.xml, in the administrative console, do the following:
|
Resolution: For all certificates that are not signed by the WebSphere root certificate for the cell, if you'd like to replace those with SHA256 certificates, that will need to be done manually. This means that any personal certificate will need to be re-created, and any CA certificate will need to be replaced with a new one from the CA.
java.security.cert.CertPathValidatorException: The revocation status of the certificate with subject (CN=myserver.ibm.com) could not be determined
I am unable to login into admin console with LDAP user and I see the following error after migrating to WebSphere Application Server v6.1 to WebSphere Application Server 7.0 or 8.0.CWPKI0022E: SSL HANDSHAKE FAILURE: A signer with SubjectDN "CN=myserver.ibm.com" was sent from target host:port "*:9043". The signer may need to be added to local trust store "/was/was/WebSphere/AppServer/profiles/dmgr01/config/cells/cellname/trust.p12" located in SSL configuration alias "CellDefaultSSLSettings" loaded from SSL configuration file "security.xml". The extended error message from the SSL handshake exception is: "PKIX path validation failed: java.security.cert.CertPathValidatorException: The revocation status of the certificate with subject (CN=myserver.ibm.com) could not be determined." |
Discussion on dwAnswers.
This is an error that results from certificate revocation evaluation and may be the result of an environment migrated existing WebSphere v6.1 server to a later version.
For WebSphere v6.1:
|
Consider the following when migrating WebSphere v6.1 to v7.0 and above:
|
So, on v6.1, if you are still using the default value of true for the com.ibm.jsse2.checkRevocation property and the default trust manager, IbmX509, and you have a certificate revocation evaluation issue with your certificate (but you don't know it), then you migrate to v7.0 or above, you will get the java.security.cert.CertPathValidatorException error.
Resolution: Set the com.ibm.jsse2.checkRevocation property to false.
In the administrative console, do the following:
- Navigate to Security > SSL certificate and key management > SSL configurations > NodedefaultSSLSetting
- Under Additional Properties, click Trust and key managers
- Under Related Items, click Trust managers
- Click IbmPKIX > Custom properties
- Do one of the following:
- If the com.ibm.jsse2.checkRevocation property does not exist, add
com.ibm.jsse2.checkRevocation=false - If the com.ibm.jsse2.checkRevocation property exists, set the value to false.
- If the com.ibm.jsse2.checkRevocation property does not exist, add
- Click OK
- Save the configuration
- Restart the application server
The com.ibm.jsse2.checkRevocation property that configures revocation checking for the JVM is set to false by default because the default WebSphere certificates used for SSL communication do not contain certificate revocation list (CRL) distribution points or Online Certificate Status Protocol (OCSP) information.
When com.ibm.jsse2.checkRevocation is set to true, the JVM will attempt to check whether the certificate being used is revoked. The revocation status can be determined in a few different ways. If the status cannot be determined the certificate can't be used.
javax.net.ssl.SSLKeyException: RSA premaster secret error
WebSphere is making an outbound SSL call to a remote server and the remote server has upgraded their certificate.On WebSphere side, I added the remote server root and intermediate signer certificates in WebSphere truststore. After the update, WebSphere log shows the following errors:
handling exception: javax.net.ssl.SSLKeyException: RSA premaster secret error SystemErr R javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated SystemErr R at com.ibm.jsse2.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:113 |
Discussion on dwAnswers.
Resolution:
To resolve this issue, you need to update your JDK to use the Unrestricted SDK JCE policy files. You must place the US_export_policy.jar and the local_policy.jar in the (was_home)/jre/lib/security/ directory of the Java™ Runtime Environment (JRE). These files can be obtained from the ibm.com® website IBM Unrestricted JCE policy files.
When you click the link for obtaining the IBM Unrestricted JCE policy files, you will be prompted to log in using your IBM ID and password. If you do not have an IBM ID and password, you will need to register. Follow the registration link on the login page.
After logging in, you will be presented with a two choices:
- Java 5.0 SR16, Java 6 SR13, Java 6 SR5 (J9 VM2.6), Java 7 SR4, Java 8 GA, and all later releases
- Files for older versions of the SDK
You need to make sure what JDK version you are using with WebSphere. For Example, if you are using Java 6 SR13 or later versions then you select FIRST option. Anything before Java 6 SR13 you need to select SECOND option.
![](/support/pages/system/files/support/swg/swgtech.nsf/0/beebcc764b72c78585258018004d97b5/Content/0.8A30.gif)
WAS_INSTALL_ROOT/java/bin/java -version |
After you download the .zip file, unpack it. Two files are included in the .zip file:
- US_export_policy.jar
- local_policy.jar
![](/support/pages/system/files/support/swg/swgtech.nsf/0/beebcc764b72c78585258018004d97b5/Content/0.8F7A.gif)
Alternatively you can also look into the Learn More section for a video that explains how to install the SDK Unrestricted SDK JCE policy files.
javax.net.ssl.SSLHandshakeException java.security.cert.CertPathValidatorException: Certificate chaining error The certificate is not trusted
WebSphere Application Server is acting as a client, making an outbound SSL call to a remote server. The remote server's root and intermediate certificates appear to be in WebSphere's truststore. The following appears in the WebSphere logs:Caused by: javax.net.ssl.SSLHandshakeException`: com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: java.security.cert.CertPathValidatorException: The certificate issued by CN=something is not trusted; internal cause is: java.security.cert.CertPathValidatorException: Certificate chaining error |
Discussion on dwAnswers.
When certificate validation is performed on the partner certificate, the JDK must follow the certificate from the user (end-entity) certificate down to the first trusted entry. If you have both the remote server's root and issuer certificates in the WebSphere truststore, you may be getting this error for one of two reasons:
- There are additional intermediate certificates missing from the WebSphere truststore
- The partner has handled its certificate incorrectly when sending it to WebSphere.
To see the certificate chain that is being sent by your server, you can use a JSSE trace or, if your endpoint is not behind a firewall, you can use a tool like the DigiCert® SSL Installation Diagnostics Tool - SSL Certificate Checker. For example, enter developer.ibm.com.
![](/support/pages/system/files/support/swg/swgtech.nsf/0/beebcc764b72c78585258018004d97b5/Content/0.9E14.gif)
It shows the following chain (> means 'is issued by'):
www.ibm.com > GeoTrust SSL CA - G3 > GeoTrust Global CA |
We can reduce this to A > B > C
If your partner sends a fully-formed certificate chain that includes all the intermediate certificates down to one that you trust, the end-entity certificate will pass validation. In this example, if you have B and C in the truststore, certificate A should pass validation. With the same truststore contents, if the partner sends A > X > Y > Z > B > C as a fully-formed chain, it will also pass validation since the trust chain can be followed down to a trusted certificate (B). However, if the chain is not fully-formed, meaning any of the X, Y, or Z certificates are missing, certificate validation for A will fail with CertPathValidatorException: Certificate chaining error
If you find that one or more intermediate certificates are missing from the chain, do one of the following:
|
If you aren't missing any intermediate certificates, next you need to check to make sure that the certificates appear in the correct order. The certificate list should start with the end-entity certificate, list the issuers in order, and terminate with the root certificate (who issues itself). If your certificate chain is A > B > C and you get back B, C, A, instead of A, B, C, you are going to get a certificate chaining error. This scenario is much less likely to happen than missing intermediate certificates.
Background: What is the SSL Certificate Chain?
javax.net.ssl.SSLException java.lang.ArrayIndexOutOfBoundsException: Array index out of range
Discussion on dwAnswers.Related APAR: IBM IV73472: LARGE PRE-MASTER SECRET GENERATED FROM 2048 BIT DH KEY NOT DIGESTED IN TLSV1 AND TLSV1.1. This problem happens when the server side uses a large DH key (e.g. 2048 bit) in a TLSv1/TLSv1.1 key exchange.
![](/support/pages/system/files/support/swg/swgtech.nsf/0/beebcc764b72c78585258018004d97b5/Content/4.A38.gif)
Workaround solutions:
|
To enable TLSv1.2 using the admin console:
|
Edit the ssl.client.props file to use TLSv1.2:
|
After making updates in the admin console and all the ssl.client.props files:
|
End user tried to act as a CA
Following error is seen when attempting to validate a server certificate chain during SSL operation:End user tried to act as a CA. |
Cause
Root CA certificate does not contain BASIC CONSTRAINTS extension.
According to the X509v3 specifications, any root CA certificate is must contain a Basic Constraints extension and the value of the CA flag must be set to TRUE. Otherwise, the certificate is deemed to be an end-entity certificate and not a CA certificate. During SSL handshake, if a server certificate chain uses a root CA certificate that does not have this extension set correctly, the handshake can fail at the client end with the "End user tried to act as a CA." message
Resolution:
- To resolve the issue, contact the CA who issued the certificate and get the server certificate chain corrected.
- If you created the CA certificate yourself using the Java 7 (or later) keytool utility, the -ext option on the -gencert command is used to specify BasicConstraints. See the keytool documentation, ORACLE® Java SE Documentation: keytool - Key and Certificate Management Tool, for more information.
- If you created the CA certificate yourself using openssl x509 command, make sure that you specify v3_ca for the -extensions parameter. See the OpenSSL x509 command documentation, Open SSL 1.1.0 manpages: x509, for more information.
- Another potential workaround is to use the IbmPKIX trust manager instead of IbmX509 trust manager at the client end. The trust manager is set in java.security file using the property "ssl.TrustManagerFactory.algorithm"
How to view the Basic Constraints extension in a keystore:
- Keytool
- Using the keytool utility from Java 7 or later, you can see the extensions attributes of a certificate. Here is the output of keytool -list -v of a CA certificate:
- <snip>
Extensions:
#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 4e d9 0b fa 00 f5 09 2e N.......
]
]
#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]
- <snip>
- iKeyman
- Using the iKeyman utility that is delivered with WebSphere, you can view the extensions attributes of a certificate in a keystore. Perform the following actions to check for BasicConstraints extensions on a certificate in a keystore:
- Start iKeyman
- Open the keystore
- Select the certificate
- Click View/Edit
- Click View Details
- In the field pane, look for Extensions. It should be near the bottom a just above Signature Algorithm.
- Check for a Basic Constraints child of Extensions.
- If Basic Constraints exists, click Value
- Check for CA:true in the Value pane
- PEM file or RFC 1421 format
- If you can emit the certificate in RFC 1421 format or have a PEM file, you can use the SSL Certificate Decoder from SSLChecker.com. You can use the keytool command (from any Java version) that is delivered with WebSphere to display certificates in RFC 1421 format using the following command:
- keytool -list -rfc -keystore (keystorename) -storepass (password)
Here is the CA certificate shown above in RFC 1421 format:- -----BEGIN CERTIFICATE-----
MIICuzCCAiSgAwIBAgIGFeBStgjcMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYDVQQGEwJKUDERMA8G
A1UECBMIS2FuYWdhd2ExDzANBgNVBAcTBllhbWF0bzEMMAoGA1UEChMDSUJNMQwwCgYDVQQLEwNU
UkwxGTAXBgNVBAMTEFNPQVAgMi4xIFRlc3QgQ0ExIjAgBgkqhkiG9w0BCQEWE21hcnV5YW1hQGpw
LmlibS5jb20wHhcNMDgwODA1MTc0NjI3WhcNMjMwODA3MTc0NjI3WjCBjDELMAkGA1UEBhMCSlAx
ETAPBgNVBAgTCEthbmFnYXdhMQ8wDQYDVQQHEwZZYW1hdG8xDDAKBgNVBAoTA0lCTTEMMAoGA1UE
CxMDVFJMMRkwFwYDVQQDExBTT0FQIDIuMSBUZXN0IENBMSIwIAYJKoZIhvcNAQkBFhNtYXJ1eWFt
YUBqcC5pYm0uY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDkwms5D3TWou/DUVRqL2Sg
kr/TV/oAJppMIikplLjMG4n5YlOkxToghbUibsrx3GywfjnBcRiZNfhe9wcRNQcWPpOABDr9pW5f
7/iXVrxB8tb6yczcX60we1QBocUi3dWkgh4QxLm5b8q6G7EZFKm49UxZuFIOo3MV/OzCjYWLbQID
AQABoyYwJDARBgNVHQ4ECgQITtkL+gD1CS4wDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQUF
AAOBgQAMwsogfHnNRGdaA128QMVVWRfZOk5cvNaMIZ8dfueib7wfqXRRe1XZUqez7XBlR2u6ViiV
AGWiA32z3NB4AWCx0PpEtzh+XWM4vhotwAoyWU+etKv8WOV4hnwRguWqe/gjdIFdzbUcz43AXCJU
0gaz7eS1W9P6YHZBJKB8MNznVg==
-----END CERTIFICATE-----
If you copy this text into the SSL Certificate decoder (including the BEGIN/END certificate lines), then click Decode, you'll see all the attributes of the certificate. At the bottom, click RAW OUTPUT. Just before the Signature Algorithm you'll see the two X509v3 extensions, one of which is the Basic Constraints extension:
- X509v3 extensions:
X509v3 Subject Key Identifier:
4E:D9:0B:FA:00:F5:09:2E
X509v3 Basic Constraints: critical
CA:TRUE
Is Perfect Forward Secrecy (PFS) supported on WebSphere Application Server?
Problem
The RSA key exchange mechanism creates a link between the server’s key pair and the session key created for each unique secure session. Thus, if an attacker is ever able to get hold of the server’s private key, they can decrypt the SSL session and any saved SSL sessions. In contrast, key exchanges that meet the requirements for Perfect Forward Secrecy do not rely on a link between the server's private key and each session key. If an attacker ever gets access to the server’s private key, the attacker cannot use the private key alone to decrypt any of the archived sessions, which is why it is called "Perfect Forward Secrecy".
Note that Perfect Forward Secrecy is not affected when RSA is used as the encryption algorithm. It is only affected when RSA is used as the key exchange algorithm.
Resolution
Both DHE and ECDHE key exchange cipher suites create session keys that only the entities involved in the SSL connection can access.
For users of WebSphere Application Server 8.5.5.16 and above:
- Enable Perfect Forward Secrecy by creating a list of custom cipher suites that only use Elliptic Curve Diffie-Hellman (ECDHE) or Diffie-Hellman Exchange (DHE) key exchange cipher suites.
- See the following video for instructions for changing the strength/custom cipher suite groups in WebSphere Application Server: https://youtu.be/dheizcFimX0
Disabling 3rd party Java agents
- In the administrative console, click , and either WebSphere application servers > server_name or WebSphere proxy servers > server_name.
- Under Server Infrastructure, expand
- Click
Note:
Was this topic helpful?
Document Information
Modified date:
12 August 2023
UID
swg21989352