Refreshing IBM Cloud Pak foundational services internal certificates

Refreshing foundational services internal certificates (IBM Cloud Pak foundational services Version 3.6.4 and later)

A common CA issuer is created from a self-signed CA certificate (certificate authority) and leaf certificates are created by individual foundational services from the common CA issuer. Cert-manager automatically refreshes individual certificates. The CA certificate that is used by foundational services has a default duration of 2 years. Cert-manager will automatically refresh the CA certificate and the ibm-cert-manager-operator will automatically refresh the leaf certificates that are created from the CA certificate. You can customize the duration of the CA certificate based on the certification rotation schedule.

Changing the duration of the CA certificate and refreshing foundational services internal certificates

Complete the following steps to change the duration of the CA certificate and then refresh the foundational services internal certificates.

Note: apiVersion: certmanager.k8s.io/v1alpha1 is deprecated. Use apiVersion: cert-manager.io/v1.

  1. Update the cs-ca-certificate yaml file to add the duration and renewBefore parameters.

     duration: 17520h
 renewBefore: 240h

    In the following example, duration value is set to two years and the renewBefore value is set to 10 days.

     apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
  name: cs-ca-certificate
  namespace: ibm-common-services
  labels:
   app.kubernetes.io/instance: ibm-cert-manager-operator
   app.kubernetes.io/managed-by: ibm-cert-manager-operator
   app.kubernetes.io/name: cert-manager
   certmanager.k8s.io/issuer-kind: Issuer
   certmanager.k8s.io/issuer-name: cs-ss-issuer
  spec:
   commonName: cs-ca-certificate
   isCA: true
   issuerRef:
    kind: Issuer
    name: cs-ss-issuer
   secretName: cs-ca-certificate-secret
   duration: 17520h
   renewBefore: 240h
  status:
   conditions:
   - lastTransitionTime: '2020-12-08T04:20:27Z'
   message: Certificate is up to date and has not expired
   reason: Ready
   status: 'True'
   type: Ready
   notAfter: '2021-03-08T04:20:27Z'

  2. Delete the secret to refresh the CA certificate.

     oc -n ibm-common-services delete secret cs-ca-certificate-secret

    All leaf clusters that are created from cs-cs-certificate will be refreshed by the ibm-cert-manager-operator.

    Plan your certification rotation schedule by making note of the default CA certificate expiration, which is 2 years from deployment or the custom duration if set. The expiration date can be obtained with the following command.

     oc -n ibm-common-services get cert cs-ca-certificate

  3. After refresh, if the monitoring console is not accessible then restart the alertmanager-ibm-monitoring-alertmanager-0, prometheus-ibm-monitoring-prometheus-0, and grafana pods.

     oc delete pod -l app=alertmanager -n ibm-common-services

     oc delete pod -l app=grafana -n ibm-common-services

     oc delete pod -l app=prometheus -n ibm-common-services

Refreshing foundational services internal certificates (IBM Cloud Pak foundational services Version 3.6.3 and earlier)

A common CA issuer is created from a self-signed CA certificate (certificate authority) and leaf certificates are created by individual foundational services from the common CA issuer. The CA certificate default duration is 90 days. You can customize this duration based on the certificate rotation schedule. Cert-manager will automatically refresh the CA certificate but the leaf certificates that are created from this CA certificate must be manually refreshed when the CA certificate is refreshed.

Complete the following steps to refresh the foundational services internal certificates.

  1. Delete the secret to refresh the CA certificate.

     oc -n ibm-common-services delete secret cs-ca-certificate-secret

  2. Refresh the leaf certificates based on the cs-ca-certificate.

    This step forces the leaf certificates to be updated with the new ca.

     mkdir secret_backup

     cd secret_backup

     oc get certs -o custom-columns=:spec.secretName,:spec.issuerRef.name --no-headers |egrep "cs-ca-clusterissuer|cs-ca-issuer" | while read secretName issuerName
 do
 oc get secret $secretName -o yaml -n ibm-common-services > secret.$secretName.yaml
 oc delete secret $secretName -n ibm-common-services
 done

    After the certificates are refreshed, all IBM Cloud Pak foundational services pods that mount these certificates will be automatically refreshed by cert-manager.

  3. Restart the auth-idp, auth-pap, auth-pdp pods.

     oc delete pod -l app=auth-idp -n ibm-common-services

     oc delete pod -l app=auth-pap -n ibm-common-services

     oc delete pod -l app=auth-pdp -n ibm-common-services

  4. Check and run the security-onboarding-job as-needed.

    1. Check the status of the security onboarding job.

        oc get jobs |grep security-onboarding

      If the job ran properly, it shows completion as 1/1. If a failure occurred, it shows 0/1.

    2. If a failure occurs, take a backup of the job and then delete the job to rerun it.

        oc get job <security-onboarding-job> -o yaml > security-onboarding-backup.yaml

        oc delete job security-onboarding

    3. Wait for the job to complete.

        oc get jobs

        oc get jobs
  NAME                       COMPLETIONS   DURATION   AGE
  iam-onboarding             1/1           4m46s      157m
  oidc-client-registration   1/1           6m45s      3h11m
  security-onboarding        0/1           14s        14s
  [root@magesh-intel-inf ~]#

  oc get jobs -w
  NAME                       COMPLETIONS   DURATION   AGE
  iam-onboarding             1/1           4m46s      158m
  oidc-client-registration   1/1           6m45s      3h11m
  security-onboarding        0/1           30s        30s
  security-onboarding        1/1           88s        88s

  5. Refresh the ibmcloud-cluster-ca-cert.

    For foundational services versions before 3.6.2, the ibmcloud-cluster-ca-cert secret must be refreshed to pick up the refreshed ca.crt. The ibm-management-ingress-operator re-creates the secret.

    If you replaced the foundational services endpoint certificate by using procedure, Replacing the foundational services endpoint certificate (version 3.6.3 and earlier), then do not perform this step.

    Run the following command to delete the secret.

     oc -n ibm-common-services delete secret ibmcloud-cluster-ca-cert

  6. Re-create the cp-console route.

    For foundational services versions before 3.6.2, the cp-console route must be deleted and re-created to use the refreshed leaf certificate route-tls (secret route-tls-secret). The ibm-management-ingress-operator re-creates the route.

    Run the following command to delete the route.

     oc delete route cp-console -n ibm-common-services

  7. (Optional) If you are using your own certificate and replaced the management-ingress certificate by using procedure, Replacing the foundational services endpoint certificate (version 3.6.3 and earlier), then you must complete the following steps to update the cp-console route. The destination ca certificate is updated based on the refresh of the foundational services internal CA certificate.

    1. Obtain the updated destination CA certificate.

        oc get -o jsonpath={.data."ca.crt"} secret icp-management-ingress-tls-secret | base64 -d > dest-ca.crt

    2. Save the generated cert, key, and ca-cert of your certificate in the same directory.

    3. Regenerate the route spec.

         oc -n ibm-common-services create route reencrypt cp-console --service=icp-management-ingress     --cert=./tls.crt     --key=./tls.key     --ca-cert=./ca.crt     --dest-ca-cert=./dest-ca.crt     --hostname=cp-console.apps.demo.cp.fyre.ibm.com     --insecure-policy='Redirect'     --dry-run='client' -o yaml  > cp-console.yaml

    4. Apply the change.

        oc -n ibm-common-services  apply -f cp-console.yaml

  8. After refresh, if the monitoring console is not accessible then restart the alertmanager-ibm-monitoring-alertmanager-0, prometheus-ibm-monitoring-prometheus-0, and grafana pods.

     oc delete pod -l app=alertmanager -n ibm-common-services

     oc delete pod -l app=grafana -n ibm-common-services

     oc delete pod -l app=prometheus -n ibm-common-services

Changing the duration of the CA certificate and refreshing foundational services internal certificates

Complete the following steps to change the duration of the CA certificate and then refresh the foundational services internal certificates.

  1. Back up the certificate by running the following command:

     oc get certificate cs-ca-certificate -n ibm-common-services > cs-ca-certificate.yaml.bakup

  2. Update the cs-ca-certificate yaml file and add the duration and renewBefore parameters.

    1. To update the cs-ca-certificate yaml file run the following command:

      oc edit certificate cs-ca-certificate -n ibm-common-services

    2. Add the duration and renewBefore parameters with the following values under the spec section in the cs-ca-certificate yaml file.

        duration: 17520h
  renewBefore: 240h

      In the following example, duration value is set to two years and the renewBefore value is set to 10 days.

        apiVersion: cert-manager.io/v1
  kind: Certificate
  metadata:
    name: cs-ca-certificate
    namespace: ibm-common-services
  labels:
    app.kubernetes.io/instance: ibm-cert-manager-operator
    app.kubernetes.io/managed-by: ibm-cert-manager-operator
    app.kubernetes.io/name: cert-manager
    certmanager.k8s.io/issuer-kind: Issuer
    certmanager.k8s.io/issuer-name: cs-ss-issuer
  spec:
    commonName: cs-ca-certificate
    isCA: true
    issuerRef:
      kind: Issuer
      name: cs-ss-issuer
  secretName: cs-ca-certificate-secret
    duration: 17520h
    renewBefore: 240h
  status:
    conditions:
    - lastTransitionTime: '2020-12-08T04:20:27Z'
    message: Certificate is up to date and has not expired
    reason: Ready
    status: 'True'
    type: Ready
    notAfter: '2021-03-08T04:20:27Z'

  3. Save the cs-ca-certificate yaml file.

  4. Delete the secret to refresh the CA certificate.

     oc -n ibm-common-services delete secret cs-ca-certificate-secret

  5. Refresh the leaf certificates based on the cs-ca-certificate.

    Include this step if you refreshed cs-ca-certificate in the previous step, or if it was already refreshed by cert-manager. This step forces the leaf certificates to be updated with the new ca.

     mkdir secret_backup

     cd secret_backup

     oc get certs -o custom-columns=:spec.secretName,:spec.issuerRef.name --no-headers |egrep "cs-ca-clusterissuer|cs-ca-issuer" | while read secretName issuerName
 do
 oc get secret $secretName -o yaml -n ibm-common-services > secret.$secretName.yaml
 oc delete secret $secretName -n ibm-common-services
 done

    After the certificates are refreshed, all IBM Cloud Pak foundational services pods that mount these certificates will be automatically refreshed by cert-manager.

  6. Restart the auth-idp, auth-pap, auth-pdp pods.

     oc delete pod -l app=auth-idp -n ibm-common-services

     oc delete pod -l app=auth-pap -n ibm-common-services

     oc delete pod -l app=auth-pdp -n ibm-common-services

  7. Check and run the security-onboarding-job as-needed.

    1. Check the status of the security onboarding job.

        oc get jobs |grep security-onboarding

      If the job ran properly, it shows completion as 1/1. If a failure occurred, it shows 0/1.

    2. If a failure occurs, take a backup of the job and then delete the job to rerun it.

        oc get job <security-onboarding-job> -o yaml > security-onboarding-backup.yaml

        oc delete job security-onboarding

    3. Wait for the job to complete.

        oc get jobs

        oc get jobs
  NAME                       COMPLETIONS   DURATION   AGE
  iam-onboarding             1/1           4m46s      157m
  oidc-client-registration   1/1           6m45s      3h11m
  security-onboarding        0/1           14s        14s
  [root@magesh-intel-inf ~]#

  oc get jobs -w
  NAME                       COMPLETIONS   DURATION   AGE
  iam-onboarding             1/1           4m46s      158m
  oidc-client-registration   1/1           6m45s      3h11m
  security-onboarding        0/1           30s        30s
  security-onboarding        1/1           88s        88s

  8. Refresh the ibmcloud-cluster-ca-cert.

    For foundational services versions before 3.6.2, the ibmcloud-cluster-ca-cert secret must be refreshed to pick up the refreshed ca.crt. The ibm-management-ingress-operator re-creates the secret.

    If you replaced the foundational services services endpoint certificate by using procedure, Replacing the foundational services endpoint certificate (version 3.6.3 and earlier), then do not perform this step.

    Run the following command to delete the secret.

     oc -n ibm-common-services delete secret ibmcloud-cluster-ca-cert

  9. Re-create the cp-console route.

    For foundational services versions before 3.6.2, the cp-console route must be deleted and re-created to use the refreshed leaf certificate route-tls (secret route-tls-secret). The ibm-management-ingress-operator re-creates the route.

    Run the following command to delete the route.

    oc delete route cp-console -n ibm-common-services

  10. (Optional) If you are using your own certificate and replaced the management-ingress certificate by using procedure, Replacing the foundational services endpoint certificate, then you must complete the following steps to update the cp-console route. The destination ca certificate is updated based on the refresh of the foundational services internal CA certificate.

    1. Obtain the updated destination CA certificate.

       oc get -o jsonpath={.data."ca.crt"} secret icp-management-ingress-tls-secret | base64 -d > dest-ca.crt

    2. Save the cert, key, and ca-cert of your certificate in the same directory. For example,

       ls -l
 total 68
 -rw-r--r-- 1 root root  2021 Oct 19 18:17 ca.crt
 -rw-r--r-- 1 root root  1168 Oct 19 18:28 dest-ca.crt
 -rw-r--r-- 1 root root  1777 Oct 19 18:18 tls.crt
 -rw-r--r-- 1 root root  1675 Oct 19 18:17 tls.key

    3. Regenerate the route spec.

        oc -n ibm-common-services create route reencrypt cp-console --service=icp-management-ingress     --cert=./tls.crt     --key=./tls.key     --ca-cert=./ca.crt     --dest-ca-cert=./dest-ca.crt     --hostname=cp-console.apps.demo.cp.fyre.ibm.com     --insecure-policy='Redirect'     --dry-run='client' -o yaml  > cp-console.yaml

    4. Apply the change.

       oc -n ibm-common-services  apply -f cp-console.yaml

  11. After refresh, if the monitoring console is not accessible then restart the alertmanager-ibm-monitoring-alertmanager-0, prometheus-ibm-monitoring-prometheus-0, and grafana pods.

    oc delete pod -l app=alertmanager -n ibm-common-services

    oc delete pod -l app=grafana -n ibm-common-services

    oc delete pod -l app=prometheus -n ibm-common-services