Troubleshooting
Problem
After an OpenShift worker node was rebooted the user receives after login
504 Gateway Time-out
The server didn't respond in time.
Cause
The Mongo pod's IP address was used in the MAS initial setup after installation. After the worker node was rebooted, the IP of the pod changed and for this reason the coreidp pod cannot reach the mongodb to authorize the user.
Diagnosing The Problem
In coreidp logs the below error messages appear:
[4/17/21 2:28:13:953 UTC] 0000003b id=00000000 org.mongodb.driver.cluster I Exception in monitor thread while connecting to server x.x.x.x:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.SocketStream.open(SocketStream.java:62)
at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:126)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:114)
at java.lang.Thread.run(Thread.java:822)
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:236)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:218)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.net.Socket.connect(Socket.java:682)
at com.ibm.jsse2.av.connect(av.java:619)
at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:59)
at com.mongodb.connection.SocketStream.open(SocketStream.java:57)
... 3 more
and
4/20/21 7:06:17:345 UTC] 00009789 id=00000000 ibm.ws.security.oauth20.plugins.custom.OauthConsentStoreImpl E CWWKS1480E: The OAuth feature encountered an error while deleting the expired OAuthConsent entries from the OAuthStore implementation. AIUOA0023E: Failed on deleteConsents for time after 1,618,902,347,345.
[4/20/21 7:06:17:345 UTC] 00009788 id=00000000 com.ibm.ws.security.oauth20.plugins.custom.OauthTokenStore E CWWKS1478E: The OAuth feature encountered an error while deleting the expired OAuthToken entries from the OAuthStore implementation. AIUOA0021E: Failed on deleteTokens for time after 1,618,902,347,345.
[4/20/21 7:06:17:345 UTC] 00009789 id=00000000 org.mongodb.driver.cluster I Cluster description not yet available. Waiting for 30000 ms before timing out
[4/20/21 7:06:17:345 UTC] 00009788 id=00000000 org.mongodb.driver.cluster I Cluster description not yet available. Waiting for 30000 ms before timing out
Resolving The Problem
Update the mongo configuration map in mas-core namespace to use the mongo service name for host parameter instead of IP. Mongo service name can be retrieved from the namespace where mongodb pod exists.
For example:
Faulty configuration map:
kind: ConfigMap
apiVersion: v1
metadata:
annotations: {}
namespace: mas-support-core
ownerReferences:
- apiVersion: config.mas.ibm.com/v1
kind: MongoCfg
name: emea-mongo-system
labels:
app.kubernetes.io/instance: support
app.kubernetes.io/managed-by: ibm-mas-cfg-mongo
app.kubernetes.io/name: ibm-mas
mas.ibm.com/instanceId: support
data:
mongo.yaml: |
hosts:
[{'host': 10.10.0.10 ,'port': 27017}]
configDb: admin
authMechanism: DEFAULT
Get service name from mongo name space:
[root@ibm.com ~]# oc get svc -n mongo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mas-mongo-ce-svc ClusterIP None <none> 27017/TCP 59d
Correct the configuration map by changing the host parameter from the used IP to above service name:
IMPORTANT: This applies for MongDB instances that are not deployed as cluster instances, they are single pod deployments type.
kind: ConfigMap
apiVersion: v1
metadata:
annotations: {}
namespace: mas-support-core
ownerReferences:
- apiVersion: config.mas.ibm.com/v1
kind: MongoCfg
name: emea-mongo-system
labels:
app.kubernetes.io/instance: support
app.kubernetes.io/managed-by: ibm-mas-cfg-mongo
app.kubernetes.io/name: ibm-mas
mas.ibm.com/instanceId: support
data:
mongo.yaml: |
hosts:
[{'host': 'mas-mongo-ce-svc' ,'port': 27017}]
configDb: admin
authMechanism: DEFAULT
If you deployed the MongoDB Community Edition as described in this document, then you need to add in the configuration map all the hostnames of the members in the MongoDB cluster. You can retrieve them using the below command:
[root@ibm.com ~]# oc get MongoDBCommunity -n mongo -o 'jsonpath={..status.mongoUri}'
mongodb://mas-mongo-ce-0.mas-mongo-ce-svc.mongo.svc.cluster.local:27017,
mas-mongo-ce-1.mas-mongo-ce-svc.mongo.svc.cluster.local:27017,
mas-mongo-ce-2.mas-mongo-ce-svc.mongo.svc.cluster.local:27017
With above hostnames, the configuration map is:
kind: ConfigMap
apiVersion: v1
metadata:
annotations: {}
namespace: mas-support-core
ownerReferences:
- apiVersion: config.mas.ibm.com/v1
kind: MongoCfg
name: emea-mongo-system
labels:
app.kubernetes.io/instance: support
app.kubernetes.io/managed-by: ibm-mas-cfg-mongo
app.kubernetes.io/name: ibm-mas
mas.ibm.com/instanceId: support
data:
mongo.yaml: |
hosts:
[{'host': 'mas-mongo-ce-0.mas-mongo-ce-svc.mongo.svc.cluster.local', 'port': 27017}, {'host': 'mas-mongo-ce-1.mas-mongo-ce-svc.mongo.svc.cluster.local', 'port': 27017}, {'host': 'mas-mongo-ce-2.mas-mongo-ce-svc.mongo.svc.cluster.local', 'port': 27017}]
configDb: admin
authMechanism: DEFAULT
If you need further assistance with this, please open a case with IBM Support.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB59","label":"Sustainability Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSRHPA","label":"IBM Maximo Application Suite"},"ARM Category":[{"code":"a8m3p000000hAeeAAE","label":"Maximo Application Suite->Core"}],"ARM Case Number":"TS005458265","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.0;8.0.1;8.1.0;8.2.0;8.3.0"}]
Was this topic helpful?
Document Information
Modified date:
15 October 2021
UID
ibm16445467