Generating custom certificates using cert-manager in a two data center deployment

Generate custom certificates using cert-manager in a two data center disaster recovery deployment on Kubernetes.

1. Set KUBECONFIG for the target cluster:

   export KUBECONFIG=<path_to_cluster_config_YAML_file>

   Example path:

   /Users/user/.kube/clusters/<cluster_name>/kube-config-<cluster_name>.yaml

2. Install cert-manager:
   1. Download v1.18.2 from https://github.com/cert-manager/cert-manager/releases/tag/v1.18.2.
   2. Install cert-manager. Do not specify a namespace:

      kubectl apply -f cert-manager.yaml

   3. Check the status of the cert-manager pods:

      kubectl -n cert-manager get pods

      Wait for cert-manager pods to enter Running 1/1 status before proceeding. Cert-manager has 3 pods in total.

3. Set up the customized external certificates:

   For quick setup, you can use the provided YAML files that are included in the helper_files, see Deploying operators and cert-manager. The YAML for the active data center (DC1) is called custom-certs-external-dc1.yaml and the YAML for the warm-standby data center (DC2) is called custom-certs-external-dc2.yaml.

   1. Open custom-certs-external-dc1.yaml and custom-certs-external-dc2.yaml in a text-editor. Find and replace each occurrence of example.com with the ingress subdomain for your API Connect deployment.
   2. Ensure that the management CR for DC1 is ready to be installed. Make a note of the hostname of the site.
   3. In custom-certs-external-dc1.yaml find spec.dnsNames
      For example, in DC1:

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: dc1-mgmt-replication
      spec:
        commonName: dc1-mgmt-replication
        secretName: dc1-mgmt-replication
        dnsNames:
        - <INSERT SITE HOSTNAME HERE. THIS WILL BE THE SAME VALUE AS IN THE MANAGEMENT CR spec.multiSiteHA.hosts.name for site 1>
        issuerRef:
          name: ingress-issuer
        usages:
        - "signing"
        - "key encipherment"
        duration: 17520h # 2 years
        renewBefore: 720h # 30 days
        privateKey:
          rotationPolicy: Always

   4. Replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
   5. Ensure that the management CR for DC2 is ready to be installed. Make a note of the hostname of the site.
   6. In custom-certs-external-dc2.yaml find spec.dnsNames and replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
   7. Ensure that the portal CR for DC1 is ready to be installed. Make a note of the hostname of the site.
   8. In custom-certs-external-dc1.yaml find spec.dnsNames
      For example, in DC1:

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: dc1-ptl-replication
      spec:
        commonName: dc1-ptl-replication
        secretName: dc1-ptl-replication
        dnsNames:
        - <INSERT SITE HOSTNAME HERE. THIS WILL BE THE SAME VALUE AS IN THE PORTAL CR spec.multiSiteHA.hosts.name for site 1>
          issuerRef:
          name: ingress-issuer
        usages:
        - "signing"
        - "key encipherment"
        duration: 17520h # 2 years
        renewBefore: 720h # 30 days
        privateKey:
          rotationPolicy: Always

   9. Replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
   10. Ensure that the portal CR for DC2 is ready to be installed. Make a note of the hostname of the site.
   11. In custom-certs-external-dc2.yaml find spec.dnsNames and replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
4. Use the following steps to synchronize the ingress-ca secrets across the data centers.
   1. On DC1 apply the file custom-certs-external-dc1.yaml:

      kubectl -n <namespace> apply -f custom-certs-external-dc1.yaml

   2. Validate that the command succeeded:

      kubectl get certificates -n <namespace>

   3. Export ingress-ca secret as a YAML file from DC1:

      kubectl -n <namespace> get secret ingress-ca -o yaml > ingress-ca.yaml

   4. Edit the ingress-ca.yaml file to remove all labels, annotations, creationTimestamp, resourceVersion, uid, and selfLink.
   5. Copy the ingress-ca.yaml from DC1 to DC2 and apply that file on DC2:

      kubectl -n <namespace> apply -f ingress-ca.yaml
      

   6. On DC2 apply the file custom-certs-external-dc2.yaml:

      kubectl -n <namespace> apply -f custom-certs-external-dc2.yaml
      

   7. Use the following commands to test that they are the same, on DC1 run:

      kubectl -n <namespace> get secrets ingress-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc1

   8. On DC2 run:

      kubectl -n <namespace> get secrets ingress-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc2

   9. To see the differences run:

      diff /tmp/ingress.pem.dc1 /tmp/ingress.pem.dc2 

      The files should be the same.
   10. To verify that the certificates are working correctly and that they are using the ingress-ca secret, get the portal-admin-client crt file:

       kubectl -n <namespace> get secrets portal-admin-client -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/admin-client.crt

   11. Use OpenSSL to verify the certificate:

       openssl verify -verbose -CAfile /tmp/ingress.pem.dc1 /tmp/admin-client.crt

       If it is valid, you should see:

       /tmp/admin-client.crt: OK