Certificate expiration warnings

IBM Spectrum Scale writes warning messages into the mmfs.log file for digital certificates that are nearing their expiration dates.

Warnings are issued for both RKM server certificates and key client certificates.
Note: To renew an expired server or client certificate, see the topic Renewing client and server certificates.

Warnings for an RKM server certificate

A warning message for an RKM server certificate that is approaching its expiration date contains the date and time of expiration and the IP address and port of the RKM server, as in the following example. In the log file this message would be printed all on one line:

2018-08-01_11:45:09.341-0400: GPFS: 6027-3732 [W] The server certificate for key
server 192.168.9.135 (port 5696) will expire at Aug 01 12:03:32 2018 EDT (-0400).
With this information you can log on to the specified RKM server and find the server certificate that is approaching expiration.

Warnings for a key client certificate

The warning message for a key client certificate that is approaching its expiration date contains the date and time of the expiration, the IP address and port of the RKM server to which the key client has a connection, the label of the client certificate, and the RKM ID. In the log file this message would be printed all on one line:
2020-11-04_13:55:07.838-0400: [W] The client certificate with label 'client1' for key server with RKM ID 'RKM1' (192.168.9.135:5696) will expire at Nov 04 16:39:59 2020 EDT (-0400).
The procedure for identifying an expiring client certificate based on the RKM server information in the error message depends on two circumstances:
  • Whether more than one key client in the cluster has a connection with the RKM server that is specified in the error message.
  • Whether the encryption environment of the cluster is configured by the simplified setup method or the regular setup method.
The following instructions assume that only one key client in the cluster has a connection with the specified RKM server:
  • Simplified method: If the encryption environment is configured by the simplified method, follow these steps:
    1. Make a note of the following information:
      • The expiration date of the client certificate from the warning message.
      • The IP address and port of the RKM server from the error message.
      • The host name of the RKM server that uses that IP address and port. Look this item up in your system information.
    2. On the command line of a node in the cluster, issue the following command to list the key clients for the RKM server:
      mmkeyserv client show -server <host_ID>
      where <host_ID> is the IP address or host name of the RKM server from Step 1.
    3. For each key client the command displays a block of information that includes the client certificate label, the host name or IP address and the port of the RKM server, and other information.
    4. This set of instructions assumes that only one key client in the cluster has a connection with the specified RKM server. Therefore, in Step 3 the command displays only one block of information. The label that is listed in this block of information is the label of the client certificate that is approaching expiration.

  • Regular method: If the encryption environment is configured by the regular method, follow these steps:
    1. Make a note of the following information:
      • The expiration date of the client certificate from the warning message
      • The IP address and port of the RKM server from the error message.
      • The host name of the RKM server that uses that IP address and port. Look this item up in your system information.
    2. On a node of the cluster that accesses encrypted files – that is, on a node that is successfully configured for encryption – open the RKM.conf file with a text editor. For more information about the RKM.conf file, see the topic Preparation for encryption.
    3. In the RKM.conf file, follow these steps:
      1. Find the stanza that contains the host name or IP address and the port of the RKM server from Step 1. This information is specified in the kmipServerURI parameter of the stanza.
      2. The client certificate label that is specified in that same stanza is the label of the client certificate that is approaching expiration.
      3. Make a note of the path of the keystore and the keystore password that are also specified in the stanza. You can use this information to open the keystore with a tool such as the openssl key-management utility and inspect the certificate.
If more than one key client in the cluster might have a connection with the RKM server that is specified in the error message, then you must identify each such key client and search its keystore to find the certificate that is approaching expiration. The following instructions are for both the simplified setup method and the regular setup method:
  1. Make a note of the expiration date of the client certificate and the IP address and port of the RKM server in the error message. Also look up the host name of the RKM server.
  2. List the stanzas of the RKM.conf file:
    • For the simplified setup method, issue the following command from the command line:
       mmkeyserv rkm show
    • For the regular setup method, open the RKM.conf file with a text editor. You must do this step on a node that is configured for encryption. For more information about the RKM.conf file, see the topic Preparation for encryption.
  3. Find the stanza or stanzas that contain the host name or IP address of the RKM server from Step 1. For each such stanza, make a note of the client certificate label, the path of the keystore file, and the password to the keystore file.
  4. Open each keystore file from Step 3 with a tool such as the openssl key-management utility. In the keystore file, find the client certificate label or labels from Step 3 and verify whether each client certificate is approaching expiration.
To renew an expired client certificate, see the topic Renewing client and server certificates.

Only certificates that are in use are checked

IBM Spectrum Scale checks certificate expiration dates only when the certificates are being used to authenticate a connection between a key client and a key server.

IBM Spectrum Scale checks the certificate expiration dates of a key client and its RKM server at regular intervals, currently every 15 minutes. The first check occurs when the key client connects with the server to obtain a master encryption key (MEK), which it stores in a local cache on the network node. Subsequent checks occur regularly as the key client periodically reconnects with the RKM server so that it can refresh the MEK in the local cache. The current refresh interval is 15 minutes.

IBM Spectrum Scale does not check the certificate expiration dates of client or server certificates that are not currently being used in this way. This category includes not-in-use client certificates in local keystores and not-in-use server certificates for RKM backup servers.

Frequency of warnings

The frequency of warnings increases as the expiration date nears, as the following table illustrates:
Table 1. Frequency of warnings
Time before expiration Frequency of warnings
More than 90 days No warnings are logged.
30 - 90 days Every seven days.
7 days - 30 days Every 24 hours.
24 hours - 7 days Every 60 minutes.
Less than 24 hours Every 15 minutes.
A first warning is issued when both of the following conditions become true:
  • At least 75 percent of the certificate validity period has passed.
  • The time that remains falls within one of the warning windows.
Subsequent warnings are issued with the frequency that is listed in the second column of the preceding table. For example, if the validity period is 30 days and begins at midnight on March 1, then the warnings are issued as shown in the following list:
  • First warning: March 22 at 12:00 noon (.75 * 30 days = 22.5 days).
  • Second warning: March 23 at 12:00 noon (7.5 days remaining).
  • Third warning: March 24 at 12:00 noon (6.5 days remaining).
  • Warnings: Every 60 minutes from March 24 at 1:00 PM until March 29 at 12:00 midnight.
  • Warnings: Every 15 minutes from March 29 at 12:15 AM until March 30 at midnight.

Limitations

This feature has the following restrictions and limitations:
  • Warnings are logged only on nodes that access encrypted files.
  • Warnings are logged only for certificates that are used to authenticate a connection between a key client and an RKM server that is still active.
  • Warning messages identify only the type of certificate (client or server) and the IP address and port of the RKM server.