Certificate expiration dates and error messages

Learn how to check the expiration dates of Remote Key Management (RKM) server certificates and key client certificates. Also, learn to recognize the error messages that report that an RKM server certificate or a key client certificate expired.

Checking the expiration dates of RKM server and key client certificates

  • If you are using the simplified setup method in IBM Spectrum Scale 5.0.3 or later, follow these steps for the RKM server certificate and the key client certificate:
    RKM server certificate
    Issue the following command:
    mmkeyserv server show ServerName
    where ServerName is the host name or IP address that you specified for the server in the mmkeyserv server add command. In the following example, the server name is hs21n62. The expiration date of the server is displayed in the line that begins KMIP Certificate Expiration:
    # mmkeyserv server show hs21n62
hs21n62.gpfs.net
Type:                         ISKLM
IPA:                          192.168.38.14
User ID:                      SKLMAdmin
REST port:                    9443
Label:                        2_hs21n62
NIST:                         on
FIPS1402:                     on
Backup Key Servers:
Distribute:                   yes
Retrieval Timeout:            60
Retrieval Retry:              3
Retrieval Interval:           10000
REST Certificate Expiration:  2035-02-01 21:35:02 (-0500)
KMIP Certificate Expiration:  2028-04-24 22:51:31 (-0400)
    Key client certificate
    Issue the following command:
    mmkeyserv client show ClientName
    where ClientName is the name that you specified for the client in the mmkeyserv client create command. In the following example, the client name is sklm4Client. The expiration date of the client is displayed in the line that begins Certificate Expiration:
    # mmkeyserv client show sklm4Client
sklm4Client
Label:                   sklm4Client
Key Server:              hs21n62.gpfs.net
Tenants:                 Newsklm4Tenant,sklm4Tenant
Certificate Expiration:  2023-03-27 10:45:10 (-0400)
  • If you are using the regular setup method, follow these steps for the RKM server certificate and the key client certificate:

    RKM server certificate
    If the RKM server is running with a self-signed certificate, follow these steps:
    1. Issue the mmsklmconfig restcert command to retrieve the server certificate. For more information, see Renewing expired client certificates.
    2. Issue the mmgskkm print command to display the contents of the server certificate. For more information, see Renewing expired client certificates.
    3. In the mmgskkm print command output, find the expiration date for the server certificate. In the following example, the expiration date is on the final line, which begins "Valid until":
      # mmsklmconfig restcert --host hs21n62 --port 5696 --prefix sklmCert --keystore sklm4Client.p12 --keypass keystorePass
# ls -ltr sklmCert0.cert
-rw-r--r--. 1 root root 1017 Jul 27 00:55 sklmCert0.cert
# mmgskkm print --cert  sklmCert0.cert
Serial number:          2f2409efce9447
SHA-256 digest:         0c9fabf65ab3bea6259af4829cf4027db1395d46a71d49631af7c2a3454ff20d
Signature:              
 788c8c9a3ec673ac7276283f6720ff4c910f9235042f2959eb37a466277d11a9f085112e28126b05c64516
 50c9595bd21ab48aabac1ac1fab4a8e945f3dfd2de12c82f57c44e13d983305c3a7ba41d8d565c9db6a545
 981c16b12af7538f85740e6d0500266cec9fc2cf4b878c7ef12d18fd10e43c0933d246ab825dc5f059c6bb
 0e82f5fabd302e661584deb63b5feb36ed603276a9684ea240874a504dada69670c0f83a9c8767e9744e24
 a24c92dd02ca1aa94c83430d748db81ed415ac4c9b3e66593b4b2f15b094ca42a1abf6e4e9b17cba21162c
 10450c9d7314ff2ae8b62c32133c749d1d9d292d6fd320837b449a7d51a798b74b3e91cf542dc623fa
Signature algorithm:    SHA256WithRSASignature
Key size:               2048
Issuer:                 CN=crypt
Subject:                CN=crypt
Valid from:             Feb 06 21:51:31 2020 EST (-0500)
Valid until:            Apr 24 22:51:31 2028 EDT (-0400)
    If the RKM server is running with a certificate chain from a CA, follow these steps:
    1. Manually copy the files of the certificate chain from the server to a location that is accessible to the key client.
      Note: If you have not already done so, save the files of the certificate chain to a secure location. Include the root certificate file, any intermediate certificate files, and the endpoint certificate file. Now, when a client certificate expires, you will not need to download the certificate chain from the server again. You can add your local copy of the files in the server certificate chain to the new client keystore. For more information, see Renewing expired client certificates.
    2. For each certificate in the chain, do the following actions:
      1. Issue the mmgskkm print command to display the contents of the certificate. The following example displays the first certificate of a chain:
        mmgskkm print --cert sklmChain0.cert
      2. In the mmgskkm print command output, find the expiration date for the server certificate. In the following example, the expiration date is on the final line, which begins "Valid until":
        # mmsklmconfig restcert --host hs21n62 --port 5696 --prefix sklmCert --keystore sklm4Client.p12 --keypass keystorePass
# ls -ltr sklmCert0.cert
-rw-r--r--. 1 root root 1017 Jul 27 00:55 sklmCert0.cert
# mmgskkm print --cert  sklmCert0.cert
Serial number:          2f2409efce9447
SHA-256 digest:         0c9fabf65ab3bea6259af4829cf4027db1395d46a71d49631af7c2a3454ff20d
Signature:              
 788c8c9a3ec673ac7276283f6720ff4c910f9235042f2959eb37a466277d11a9f085112e28126b05c64516
 50c9595bd21ab48aabac1ac1fab4a8e945f3dfd2de12c82f57c44e13d983305c3a7ba41d8d565c9db6a545
 981c16b12af7538f85740e6d0500266cec9fc2cf4b878c7ef12d18fd10e43c0933d246ab825dc5f059c6bb
 0e82f5fabd302e661584deb63b5feb36ed603276a9684ea240874a504dada69670c0f83a9c8767e9744e24
 a24c92dd02ca1aa94c83430d748db81ed415ac4c9b3e66593b4b2f15b094ca42a1abf6e4e9b17cba21162c
 10450c9d7314ff2ae8b62c32133c749d1d9d292d6fd320837b449a7d51a798b74b3e91cf542dc623fa
Signature algorithm:    SHA256WithRSASignature
Key size:               2048
Issuer:                 CN=crypt
Subject:                CN=crypt
Valid from:             Feb 06 21:51:31 2020 EST (-0500)
Valid until:            Apr 24 22:51:31 2028 EDT (-0400)
    Key client certificate
    If the client certificate file is available, issue the mmgskkm print command to display the contents of the client certificate. In the following example, the expiration date is on the final line, which begins "Valid until":
    # mmgskkm print --cert dsm64Client.cert
Serial number:          3c4e5eae7b9785ec
SHA-256 digest:         2f97b01a1ac82b05cbdc1ac9dfe925cdb03afbdd196d8e312068923c08ceaa36
 Signature:              
 a51f8c10d5970e96eda2b8394b334d51886b827d05585edf222c881410e5cbceff4023281f5b5b9aebbb4b
 357afb56909b9d070c9fb971c5fdf5436d22526e8903a7f663da8f7380c85b31e23f48e551c9c366edc3bc
 331f6b146c6908e50aca0a69432f1cd5f130eec5afaeeb2ef85bdd9d474345719bfc2c82c23bf96066f4ec
 80d3ea43986297ace819435b547d7685c81b786d6ffacd2b0a6f6842502b5641f44dbf8acf90cb82e59595
 d1f5bb83466f7ce573d290eab76e2cbc9401017f0155a0150a7c12442b68aa4ec403f0f448ff3112039721
 85a3f39932aea84847266b9931156660bc3286153d4064e2eda29068661ef298c1cd6a3735f50a02e7
Signature algorithm:    SHA256WithRSASignature
Key size:               2048
Issuer:                 CN=dsm64Client
Subject:                CN=dsm64Client
Valid from:             Mar 08 18:08:15 2020 EDT (-0400)
Valid until:            Mar 09 17:08:15 2021 EST (-0500)
    If the client certificate file is not available, issue the openssl command to extract the client certificate from the client keystore and show the expiration date. In the following example, the client keystore file is SKLM.p12:
    # openssl pkcs12 -in SKLM.p12 -nodes | openssl x509 -noout -enddate
Enter Import Password:
MAC verified OK
notAfter=Jun 12 04:10:16 2028 GMT
    For more information, see OpenSSL.

Error message for an expired RKM server certificate

When the certificate of an RKM server expires, IBM Spectrum Scale can no longer retrieve master encryption keys (MEKs) from the server. The result is that attempts to create, open, read, or write encrypted files fail with an "Operation not permitted" error. Each time that an error occurs, IBM Spectrum Scale writes error messages like the following ones to the /var/adm/ras/mmfs.log.latest log file:
[W] The key server sklm1 (port 5696) had a failure and will be 
    quarantined for 1 minute(s).
[E] Unable to create encrypted file testfile.enc (inode 21260, 
    fileset 0, file system gpfs1).
[E] Key 'KEY-uuid:sklm1' could not be fetched. Bad certificate.

Error message for an expired key client certificate

IBM Spectrum Scale checks the status of a key client certificate each time it loads a keystore. It loads a keystore whenever a file system is mounted, or a new policy is applied, or an RKM.conf configuration file is explicitly loaded with the tsloadikm run command.

When IBM Spectrum Scale detects an expired client certificate, it writes one or more of the following error messages. The messages are written to the /var/adm/ras/mmfs.log.latest log file or to the console or to both, depending on the action that you took just before the problem occurred.
[E] Error while validating policy 'policy.enc': rc=778: 
While parsing file '/var/mmfs/etc/RKM.conf':
[E] Certificate with label 'GPFSlabel' for backend 'sklm2' has expired.