Troubleshooting
Problem
After a new certificate is generated (such as a certificate from a certificate authority (CA)) and the new keystore is applied, errors start occurring indicating that the key (certificate) in the keystore cannot be recovered. Restarts of the IBM® WebSphere® Application Server instance can also fail.
Symptom
Errors seen upon restart, or otherwise during the events, indicate that the key, or certificate, cannot be recovered. The errors can alternatively indicate the keystore password is incorrect.
The errors could be embedded within other operations, as it depends which component tried to access the keystore at a particular time. Messages can also differ on WebSphere Application Server and WebSphere Liberty
The main message to look for is "java.security.UnrecoverableKeyException: Cannot recover key", or simply just "Cannot recover key".
If you are having issues that are not specifically the error "Cannot recover key", you may be running into a different issue described in this other technote:
Troubleshooting: Unable to open PKCS12 keystores due to an UnrecoverableKeyException
Examples
- ORBX0390E: Cannot create listener thread. Exception=[ org.omg.CORBA.INTERNAL: CAUGHT_EXCEPTION_WHILE_CONFIGURING_SSL_SERVER_SOCKET, Exception=com.ibm.websphere.ssl.SSLException: java.security.UnrecoverableKeyException: Cannot recover key vmcid: 0x31415926 minor code: 77 completed: No - received while attempting to open server socket on port 31415].
- CWWSS5312E: The Application Server cannot retrieve the 'myCert' key from the '/var/keystores/key.jks' keystore. The following exception occurred: java.security.UnrecoverableKeyException: Cannot recover key
Enabling the MustGather to capture TLS traces provides more details on the errors.
Other components' tracing, or FFDC files, might reveal further details beyond what the basic error message shows.
More Examples
- [3/14/15 3:14:15:000 UTC] 00000001 AbstractJSSEP 3 Cannot recover key: invalid password for file '/var/keystores/key.jks'
[3/14/15 3:14:15:000 UTC] 00000001 AbstractJSSEP 3 Exception caught during init, java.security.UnrecoverableKeyException: Cannot recover key
[3/14/15 3:14:15:020 UTC] 00000001 JSSEHelper < The following exception occurred in getSSLContext(). Exit
java.security.UnrecoverableKeyException: Cannot recover key
at com.ibm.crypto.provider.C.recover(Unknown Source)
at com.ibm.crypto.provider.JavaKeyStore.engineGetKey(Unknown Source)
at java.security.KeyStore.getKey(KeyStore.java:1034)
at com.ibm.jsse2.az.<init>(az.java:74)
at com.ibm.jsse2.ah$a.engineInit(ah$a.java:19)
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:11)
at com.ibm.ws.ssl.provider.AbstractJSSEProvider.getKeyTrustManagers(AbstractJSSEProvider.java:552)
at com.ibm.ws.ssl.provider.AbstractJSSEProvider.generateNewSSLContext(AbstractJSSEProvider.java:220)
at com.ibm.ws.ssl.provider.AbstractJSSEProvider.getSSLContext(AbstractJSSEProvider.java:202)
at com.ibm.websphere.ssl.JSSEHelper.getSSLContext(JSSEHelper.java:763) - UnrecoverableKeyException encountered. Key password is probably incorrect.
- CWPKI0813E: Error while trying to initialize the keymanager for the keystore [/var/keystores/key.jks]. The private key password is not correct or the keystore has multiple private keys with different passwords. This keystore can not be used for SSL. Exception message is: [Cannot recover key].
Cause
The problem has several distinct root cause, including the following:
- The keystore file is corrupted (such as a zero byte file)
- The password for the keystore is incorrect
- The certificate key has a key size larger than one the JVM currently supports
- The password for the key (certificate), also known as a passphrase, does not match the keystore password
Notably, the keytool command that comes with the IBM SDK doesn't support generating PKCS12 formatted keystores with keys having different passwords (but JKS formatted keystores don't have this limitation with keytool).
It is possible that other external tools can create such a keystore with a different password for the keys.
Resolving The Problem
Keystore Password Incorrect
If there is a problem opening up the keystore, chances are the password you are supplying is incorrect.
The password for the keystore might not match the stored password in the configuration, requiring updates to either password so they match again.
- Testing the keystore password
- To test opening a keystore with a password, you can run the keytool -list command and supply the password. If it works, a list of certificates in the keystore is displayed.
- JAVA_HOME/jre/bin/keytool -list -keystore /path/to/key.jks -storetype JKS -storepass KEYSTORE_PASSWORD
- Alternatively, if the keystore is loaded into WebSphere Application Server traditional, you can also check the password known to WebSphere Application Server works by trying to open the keystore and viewing the personal or signer certificates.
- To test opening a keystore with a password, you can run the keytool -list command and supply the password. If it works, a list of certificates in the keystore is displayed.
- Updating the configuration's password
- WebSphere Application Server traditional
To update the password stored by WebSphere Application Server traditional, and not change the current password of the keystore:- In the admin console, navigate to Security > SSL Certificate and Key Management > Keystores and Certificates > KEYSTORE_NAME
- Update the password fields with the new password
- Press OK (or Apply)
- Save the changes (and synchronize if applicable)
- WebSphere Liberty
If you need to update the password stored by WebSphere Liberty, edit the server.xml (or other XML configuration file) containing the <keyStore> element matching the keystore file.- You can obtain encoded passwords on WebSphere Liberty with the securityUtility.sh command.
- WebSphere Application Server traditional
- Changing the keystore password
- If you are required to change the keystore password, recommend the keytool -storepasswd command. The -all argument is optional, and is briefly expanded upon later in the document.
JAVA_HOME/jre/bin/keytool -storepasswd -new NEW_KEYSTORE_PASSWORD -storetype PKCS12 -keystore /path/to/key.p12 -storepass KEYSTORE_PASSWORD
The command does not update the keystore password stored in the appserver's configuration.
- If you are required to change the keystore password, recommend the keytool -storepasswd command. The -all argument is optional, and is briefly expanded upon later in the document.
- Changing both configuration and keystore passwords
- WebSphere Application Server traditional
There is a Change Password button that changes both the keystore password and the configuration stored password. A requirement for this is that the password currently in the configuration is valid against the keystore. The usefulness is if the keystore password is correct but you are unable to change the key's password (passphrase), as this task can make the keystore password match the key's password.- For WebSphere Application Server traditional, if the keystore is present in Security > SSL Certificate and Key Management > Keystores and Certificates, another option is to use the Change Password button to update the keystore's password. The change here updates both the keystore's password and the stored password within the WebSphere Application Server configuration.
- Changes to the password normally take effect immediately after saving, although you might still have to synchronize the node, if applicable.
- WebSphere Liberty
As explained earlier, manually changing the server.xml configuration and updating the keystore's password (keytool for example) would allow the configuration and keystore passwords to both be updated.
- WebSphere Application Server traditional
Certificate Key Size Issue
If the key size is not supported by the JVM (such as the larger size of 4096), these errors could also occur.
- Try installing, or otherwise configuring, the unrestricted policy files to resolve the problem. Normally the unlimited policy files are enabled by default.
- Specifically, if there are policy JAR files residing in the JAVA_HOME/jre/lib/security directory, those JAR files are loaded instead of the ones normally found in the JAVA_HOME/jre/lib/security/policy/unlimited subdirectory.
- The policy files are applicable per IBM SDK installation, and require a restart of the application servers to take effect.
Key Password Mismatch
If the keystore is accessible, yet the problems remain in loading keys from the keystore, then the problem is due to the passwords for the key (certificate) and keystore being different.
There are several methods to make the key password and keystore password matching again, including the following:
- Contact your CA to regenerate your certificate (usually with a certificate request) with the key and keystore passwords matching.
- It's possible if your CA supplied a keystore with the private key and certificate chain, that the CA can make that adjustment and send a new keystore with the updates.
- Or, run a command to alter the password for the keys to match the keystore password.
- JAVA_HOME/jre/bin/keytool -keypasswd -all -keypass OLD_PASSPHRASE -new KEYSTORE_PASSWORD -storetype JKS -keystore /path/to/keystore.jks -storepass KEYSTORE_PASSWORD
- The -new and -storepass arguments are intentionally set to the same value (the argument -new represents the new key password or passphrase).
- PKCS12 keystores cannot have the key password (passphrase) different from the keystore password when keytool is run, resulting in an error message:
keytool error (likely untranslated): java.lang.UnsupportedOperationException: -keypasswd commands not supported if -storetype is PKCS12
- JAVA_HOME/jre/bin/keytool -keypasswd -all -keypass OLD_PASSPHRASE -new KEYSTORE_PASSWORD -storetype JKS -keystore /path/to/keystore.jks -storepass KEYSTORE_PASSWORD
- Or, run a command to alter the keystore password.
- JAVA_HOME/jre/bin/keytool -storepasswd -all -new NEW_PASSWORD -storetype JKS -keystore /path/to/keystore.jks -storepass KEYSTORE_PASSWORD
- The -all argument is IBM SDK specific and optional. The argument allows the key passwords to be updated as well.
- JAVA_HOME/jre/bin/keytool -storepasswd -all -new NEW_PASSWORD -storetype JKS -keystore /path/to/keystore.jks -storepass KEYSTORE_PASSWORD
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CdL1AAK","label":"WebSphere Application Server traditional-All Platforms-\u003ESecurity-\u003ESSL-\u003ESSL - Certificates"}],"ARM Case Number":"TS010712762","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSAW57","label":"WebSphere Application Server Network Deployment"},"ARM Category":[{"code":"a8m50000000CdL1AAK","label":"WebSphere Application Server traditional-All Platforms-\u003ESecurity-\u003ESSL-\u003ESSL - Certificates"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSD28V","label":"WebSphere Application Server Liberty Core"},"ARM Category":[{"code":"a8m3p000000F7yQAAS","label":"IBM WebSphere Liberty-All Platforms-\u003ELiberty SSL"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
08 June 2023
UID
ibm16826703