Portal sites do not work after failover in 2DCHA for OVA/VMware (and any form factor if the portal CR names are different).

Resolving The Problem

The workaround for the bug is as follows:

--- /opt/ibm/bin/cluster_funcs.inc	2023-01-12 11:06:28.982364359 +0000
+++ /opt/ibm/bin/cluster_funcs.inc	2023-01-12 10:29:41.412899722 +0000
@@ -96,9 +96,15 @@
 			GEN_SERVER=1
 		fi
 
-		if ! openssl x509 -in /etc/mysql/certs/server-cert.pem -text | grep -q "CN = $PORTAL_DB_SERVICE_NAME"
+		local DB_CN=$PORTAL_DB_SERVICE_NAME
+		if [[ -n "${PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME:-}" ]]
 		then
-			echo "Must regenerate server cert and key as it does not contain CN = $PORTAL_DB_SERVICE_NAME"
+			DB_CN=$DB_CN-$PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
+		fi
+
+		if ! openssl x509 -in /etc/mysql/certs/server-cert.pem -text | grep -q "CN = $DB_CN"
+		then
+			echo "Must regenerate server cert and key as it does not contain CN = $DB_CN"
 			GEN_SERVER=1
 		fi
 	fi
@@ -114,15 +120,15 @@
 		distinguished_name = dn
 
 		[ dn ]
-		CN=$PORTAL_DB_SERVICE_NAME
+		CN=$DB_CN
 
 		[ req_ext ]
 		subjectAltName = @alt_names
 
 		[ alt_names ]
-		DNS.1 = $PORTAL_DB_SERVICE_NAME
-		DNS.2 = $DEVPORTAL_FQDN
-		DNS.3 = $USE_IP
+		DNS.1 = $DEVPORTAL_FQDN
+		DNS.2 = $USE_IP
+		DNS.3 = $PORTAL_DB_SERVICE_NAME
 EOF
 
 		cat > /tmp/x509_details.txt <<- EOF
@@ -137,6 +143,16 @@
 		DNS.3 = $USE_IP
 EOF
 
+		if [[ -n "${PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME:-}" ]]
+		then
+			cat >> /tmp/csr_details.txt <<- EOF
+			DNS.4 = $PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
+EOF
+			cat >> /tmp/x509_details.txt <<- EOF
+			DNS.4 = $PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
+EOF
+		fi
+
 		openssl req -batch -new -newkey rsa:2048 -days 365000 -nodes -keyout /etc/mysql/certs/server-key-non-rsa.pem -out /etc/mysql/certs/server-req.pem -sha256 -config /tmp/csr_details.txt
 
 		openssl rsa -in /etc/mysql/certs/server-key-non-rsa.pem -out /etc/mysql/certs/server-key.pem

The provided patch must then be applied to each local db pod in both DC's by doing the following:

VMware:

If your form factor is VMware then there are 3 extra steps before running the commands listed in the "VMware/Kubernhetes/OCP" section:

• Copy cluster_funcs.inc.patch to the apicadm user's home dir on a VM in DC1 (with scp or similar).
• ssh into that VM in DC1
• Execute: sudo -i

• Execute: cat ~apicadm/cluster_funcs.inc.patch | kubectl exec -i <ptl>-db-0 -c db -- patch -p0
  • NB: ~apicadm/ is the correct path for VMware users. Replace this with the download path of cluster_funcs.inc.patch if your are on other form factors
  • Where <ptl> is the portal CR name + siteName (possibly truncated)
  • Repeat  for every local db pod. NB: The local pods ready state will be 2/2, not 1/1
• Repeat all steps for DC2

After applying the provided patch:

VMware:

• In the apicup project directory create file '<ptl-subsys-name>-extra-values.yaml' with following contents (where <ptl-subsys-name> is the unique name of the DC):

  spec:
    template:
    - containers:
      - env:
        - name: SKIP_RSYNC
          value: "1"
        - name: PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
          value: "<portal-db-service-for-the-other-dc>"
        name: db
      name: db
  

  • where <portal-db-service-for-the-other-dc> is the name of the db service for the portal in the other DC. You can get this value by running the following command in the other DC (note you will have to ssh into a portal VM and run "sudo -i" first):
    • kubectl exec <ptl>-db-0 -- bash -c 'echo $PORTAL_DB_SERVICE_NAME'
      • Where <ptl> is the portal CR name + siteName (possibly truncated)
• Execute: apicup subsys set <ptl-subsys-name> extra-values-file <ptl-subsys-name>-extra-values.yaml
  • where <ptl-subsys-name> is the subsys name of your portal in DC1.
• Execute: apicup subsys install <ptl-subsys-name>
  • where <ptl-subsys-name> is the subsys name of your portal in DC1.
• Wait for the db pods in DC1 to restart and go ready and then repeat the steps for DC2.

• Add the following section to your portal CR (Kubernetes) or top-level CR (OCP/CP4i):

  spec:
    template:
    - containers:
      - env:
        - name: SKIP_RSYNC
          value: "1"
        - name: PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
          value: "<portal-db-service-for-the-other-dc>"
        name: db
      name: db
  

  • where <portal-db-service-for-the-other-dc> is the name of the db service for the portal in the other DC. You can get this value by running the following command in the other DC:
    • kubectl exec <ptl>-db-0 -- bash -c 'echo $PORTAL_DB_SERVICE_NAME'
      • Where <ptl> is the portal CR name + siteName (possibly truncated)
• Wait for the db pods in DC1 to restart and go ready and then repeat the steps for DC2.

IMPORTANT NOTE:

VMware:

When you next upgrade, you must remove the extra-values file (By setting the value of extra-values-file to an empty string "") before you run the installation command to upgrade to the new version.

Kubernetes/OCP/CP4i:

When you next upgrade, you must remove the above template section by editing the portal CR (kubernetes) or the top-level CR (OCP/CP4i) before you run the installation command to upgrade to the new version.