Troubleshooting
Problem
When trying to access the portal sites, after a failover, the following error message is shown in the browser:
'The website encountered an unexpected error. Please try again later.'
APAR number for this issue: LI82863, which is fixed in v10.0.5.2-ifix1 and is also targeted for an upcoming fixpack v10.0.5.3.Resolving The Problem
The workaround for the bug is as follows:
Create a file named 'cluster_funcs.inc.patch' and paste the following content into it.
--- /opt/ibm/bin/cluster_funcs.inc 2023-01-12 11:06:28.982364359 +0000
+++ /opt/ibm/bin/cluster_funcs.inc 2023-01-12 10:29:41.412899722 +0000
@@ -96,9 +96,15 @@
GEN_SERVER=1
fi
- if ! openssl x509 -in /etc/mysql/certs/server-cert.pem -text | grep -q "CN = $PORTAL_DB_SERVICE_NAME"
+ local DB_CN=$PORTAL_DB_SERVICE_NAME
+ if [[ -n "${PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME:-}" ]]
then
- echo "Must regenerate server cert and key as it does not contain CN = $PORTAL_DB_SERVICE_NAME"
+ DB_CN=$DB_CN-$PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
+ fi
+
+ if ! openssl x509 -in /etc/mysql/certs/server-cert.pem -text | grep -q "CN = $DB_CN"
+ then
+ echo "Must regenerate server cert and key as it does not contain CN = $DB_CN"
GEN_SERVER=1
fi
fi
@@ -114,15 +120,15 @@
distinguished_name = dn
[ dn ]
- CN=$PORTAL_DB_SERVICE_NAME
+ CN=$DB_CN
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
- DNS.1 = $PORTAL_DB_SERVICE_NAME
- DNS.2 = $DEVPORTAL_FQDN
- DNS.3 = $USE_IP
+ DNS.1 = $DEVPORTAL_FQDN
+ DNS.2 = $USE_IP
+ DNS.3 = $PORTAL_DB_SERVICE_NAME
EOF
cat > /tmp/x509_details.txt <<- EOF
@@ -137,6 +143,16 @@
DNS.3 = $USE_IP
EOF
+ if [[ -n "${PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME:-}" ]]
+ then
+ cat >> /tmp/csr_details.txt <<- EOF
+ DNS.4 = $PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
+EOF
+ cat >> /tmp/x509_details.txt <<- EOF
+ DNS.4 = $PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME
+EOF
+ fi
+
openssl req -batch -new -newkey rsa:2048 -days 365000 -nodes -keyout /etc/mysql/certs/server-key-non-rsa.pem -out /etc/mysql/certs/server-req.pem -sha256 -config /tmp/csr_details.txt
openssl rsa -in /etc/mysql/certs/server-key-non-rsa.pem -out /etc/mysql/certs/server-key.pem
The provided patch must then be applied to each local db pod in both DC's by doing the following:
VMware:
If your form factor is VMware then there are 3 extra steps before running the commands listed in the "VMware/Kubernhetes/OCP" section:
- Copy cluster_funcs.inc.patch to the apicadm user's home dir on a VM in DC1 (with scp or similar).
- ssh into that VM in DC1
- Execute: sudo -i
- Execute: cat ~apicadm/cluster_funcs.inc.patch | kubectl exec -i <ptl>-db-0 -c db -- patch -p0
- NB: ~apicadm/ is the correct path for VMware users. Replace this with the download path of cluster_funcs.inc.patch if your are on other form factors
- Where <ptl> is the portal CR name + siteName (possibly truncated)
- Repeat for every local db pod. NB: The local pods ready state will be 2/2, not 1/1
- Repeat all steps for DC2
After applying the provided patch:
VMware:
- In the apicup project directory create file '<ptl-subsys-name>-extra-values.yaml' with following contents (where <ptl-subsys-name> is the unique name of the DC):
spec: template: - containers: - env: - name: SKIP_RSYNC value: "1" - name: PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME value: "<portal-db-service-for-the-other-dc>" name: db name: db
- where <portal-db-service-for-the-other-dc> is the name of the db service for the portal in the other DC. You can get this value by running the following command in the other DC (note you will have to ssh into a portal VM and run "sudo -i" first):
- kubectl exec <ptl>-db-0 -- bash -c 'echo $PORTAL_DB_SERVICE_NAME'
- Where <ptl> is the portal CR name + siteName (possibly truncated)
- kubectl exec <ptl>-db-0 -- bash -c 'echo $PORTAL_DB_SERVICE_NAME'
- where <portal-db-service-for-the-other-dc> is the name of the db service for the portal in the other DC. You can get this value by running the following command in the other DC (note you will have to ssh into a portal VM and run "sudo -i" first):
- Execute: apicup subsys set <ptl-subsys-name> extra-values-file <ptl-subsys-name>-extra-values.yaml
- where <ptl-subsys-name> is the subsys name of your portal in DC1.
- Execute: apicup subsys install <ptl-subsys-name>
- where <ptl-subsys-name> is the subsys name of your portal in DC1.
- Wait for the db pods in DC1 to restart and go ready and then repeat the steps for DC2.
- Add the following section to your portal CR (Kubernetes) or top-level CR (OCP/CP4i):
spec: template: - containers: - env: - name: SKIP_RSYNC value: "1" - name: PORTAL_DB_SERVICE_NAME_REMOTE_NO_SITENAME value: "<portal-db-service-for-the-other-dc>" name: db name: db
- where <portal-db-service-for-the-other-dc> is the name of the db service for the portal in the other DC. You can get this value by running the following command in the other DC:
- kubectl exec <ptl>-db-0 -- bash -c 'echo $PORTAL_DB_SERVICE_NAME'
- Where <ptl> is the portal CR name + siteName (possibly truncated)
- kubectl exec <ptl>-db-0 -- bash -c 'echo $PORTAL_DB_SERVICE_NAME'
- where <portal-db-service-for-the-other-dc> is the name of the db service for the portal in the other DC. You can get this value by running the following command in the other DC:
- Wait for the db pods in DC1 to restart and go ready and then repeat the steps for DC2.
IMPORTANT NOTE:
VMware:
When you next upgrade, you must remove the extra-values file (By setting the value of extra-values-file to an empty string "") before you run the installation command to upgrade to the new version.
Kubernetes/OCP/CP4i:
When you next upgrade, you must remove the above template section by editing the portal CR (kubernetes) or the top-level CR (OCP/CP4i) before you run the installation command to upgrade to the new version.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
01 February 2023
UID
ibm16855787