Troubleshooting
Problem
An attempt to load the "Keystores and Certificates" panel (Security > SSL Certificate and Key Management > Keystores and Certificates) in the admin console, instead a blank page is displayed. It's possible other admin console pages do not load (usually if that panel is trying to load keystores) in a similar manner. It's also possible that saving changes on panels that also list keystores (such as an SSL Configuration) will fail.
Symptom
The scenario where this failure occurs is if you delete a webserver profile before WebSphere Application Server 8.5.5.13 and then upgrade the application server to a later version. The problem is that the security.xml file is now corrupted, and the upgrade won't retroactively fix this in the profile. You may not even realize there's a problem until you have to edit the keystores or SSL configurations in the admin console as the panels will be blank.
In addition to the blank admin console page, examining the SystemOut.log (or other appserver logs) reveals a NullPointerException, usually in a "caused by" stack.
Caused by: java.lang.NullPointerException
at com.ibm.ws.console.security.SecurityUtil.populateObjectList(SecurityUtil.java:791)
at com.ibm.ws.console.security.SSLConfig.SSLConfigCollectionAction.execute(SSLConfigCollectionAction.java:188)
... 33 more
Cause
The most common reason for the issue is that the Management Scope for one of the keystores is not present anymore. The issue can occur after deletion of a server or webserver, including the CMSKeyStore (which is technically associated with a WebSphere Plug-in, but is usually configured when a webserver is created). It's also possible for the keystore referenced to not exist anymore on the filesystem.
If there were any recent changes to the servers, specifically a server or webserver profile was removed at some point, the associated keystores might still exist in the configuration. However, the attribute referencing the Management Scope is missing, causing the issue.
Resolving The Problem
The APAR PI80170 can help avoid issues when servers or webserver instances are deleted. However, if the instance was removed before application of the APAR (or upgrading the WebSphere Application Server to a version that includes this APAR), the issues may still occur.
Restoring to a known working backup is highly recommended.
Workaround
Recommend taking a backup of the profiles before proceeding.
All keystores are associated with a management scope in the security.xml (managementScope="SCOPE_NAME" in the element <keyStores>), and the issue occurs when the managementScope attribute is completely missing from the <keyStores> element. The security.xml also has a list of elements named <managementScopes> that are analogous to the attribute in the <keyStores> element, but for the affected keystore, there won't be a management scope element listed, as it would have been deleted when the server or webserver instance was removed.
To fix the problem manually:
- Locate the security.xml file: profile_root/config/cells/cell_name/security.xml
- For a deployment manager environment, it is located in the deployment manager's profile.
Changing any other profile's security.xml file will result in the changes being synchronized with the copy hosted by the deployment manager. - For a standalone appserver environment, it's located in the appserver's profile
- For a deployment manager environment, it is located in the deployment manager's profile.
- Examine the security.xml file and locate which <keyStores> element is missing its managementScope attribute. Usually it will be a keystore from a known deleted server or webserver instance.
For a webserver, that would be one of the CMSKeyStore entries. - Then delete (or comment) the identified line, and save the file.
- If on a deployment manager environment, make sure to synchronize the nodes.
- Then restart the appserver instances.
Example <keyStores> entries
This invalid entry is missing the managementScope attribute, and should be commented out or deleted as per the previous steps.
<keyStores xmi:id="KeyStore_14159265358"
name="CMSKeyStore"
password="**********"
provider="IBMCMSProvider"
location="${CONFIG_ROOT}/cells/myCell/nodes/myNode/servers/mywebserver1/plugin-key.kdb"
type="CMSKS"
fileBased="true"
createStashFileForCMS="true"
description="CMSKeyStore for web server mywebserver1."
/>
This working entry contains a managementScope attribute
<keyStores xmi:id="KeyStore_314159265359"
name="CMSKeyStore"
password="**********"
provider="IBMCMSProvider"
location="${CONFIG_ROOT}/cells/myCell/nodes/myOtherNode/servers/mywebserver2/plugin-key.kdb"
type="CMSKS"
fileBased="true"
createStashFileForCMS="true"
description="CMSKeyStore for web server mywebserver2."
managementScope="ManagementScope_3263827"
/>
NOTE: Disabling administrative security does not restore access to this panel.
IBM Software Support can be engaged to help you resolve this issue.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CcyMAAS","label":"WebSphere Application Server traditional-All Platforms-\u003ESecurity-\u003ESSL"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
27 December 2023
UID
ibm16824889