Troubleshooting
Problem
wd-discovery-ranker-rest keeps getting on crash loop back off state, Watson Discovery keeps in InProgress status. From inside the pod, there is a JIT COMPILER CRASH WITH VMSTATE=0x00040000
Steps to reproduce
- Install CP4D from IBM Maximo curated catalog
- Install Watson Discovery following Maximo procedure.
- Wait for all deployments to complete
- wd-discovery-ranker-rest should be the only deployment that is pending to complete
Symptom
Watson Discovery is on Ready state in IBM ROKS cluster but not all deployments are healthy.
Cause
This issue seems to be only relevant for IBM ROKS cluster. Issue has been originally opened against IBM Cloud Pak for Data for data team, however, IBM Cloud Pak for Data team found the problem was with the image for ranker-rest.
Environment
IBM Cloud Pak for Data version 4.6.6.
Diagnosing The Problem
Watson Discovery Ranker Rest Logs:
Unhandled exception
Type=Segmentation error vmState=0x00040000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F3C24E10830 Handler2=00007F3C246BCCD0 InaccessibleAddress=00007F3BE1BC2C80
RDI=0000000000014200 RSI=0000000000C64130 RAX=0000000000000000 RBX=0000000000000010
RCX=00007F3C22396D90 RDX=0000000000000000 R8=00007F3C2001B990 R9=0000000000000000
R10=0000000000000000 R11=00007F3BE1BC2C80 R12=0000000000000001 R13=0000000000000008
R14=00007F3C25D51D00 R15=00007F3C24EDBAF4
RIP=00007F3BE1BC2C80 GS=0000 FS=0000 RSP=00007F3C25D51B18
EFlags=0000000000010246 CS=0033 RBP=00007F3C25D51B20 ERR=0000000000000014
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=00007F3BE1BC2C80
xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1 6e6f697372655672 (f: 1919243904.000000, d: 9.083668e+223)
xmm2 00007f3c25d51fd0 (f: 634724288.000000, d: 6.911796e-310)
xmm3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm4 43e0000000000000 (f: 0.000000, d: 9.223372e+18)
xmm5 000000003dae8a3b (f: 1034848832.000000, d: 5.112833e-315)
xmm6 0000000048bb7560 (f: 1220244864.000000, d: 6.028811e-315)
xmm7 000000000003a660 (f: 239200.000000, d: 1.181805e-318)
xmm8 0000000000c69ca0 (f: 13016224.000000, d: 6.430869e-317)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000047330713 (f: 1194526464.000000, d: 5.901745e-315)
xmm14 00000000490f0490 (f: 1225720960.000000, d: 6.055866e-315)
xmm15 000000004764d420 (f: 1197790208.000000, d: 5.917870e-315)
Target=2_90_20230313_47323 (Linux 4.18.0-477.27.1.el8_8.x86_64)
CPU=amd64 (32 logical CPUs) (0x1f78997000 RAM)
Watson Discovery deployments:
$ oc get deployment -n ibm-cpd|egrep "wd-|NAME"
NAME READY UP-TO-DATE AVAILABLE AGE
wd-discovery-cnm-api 1/1 1 1 5h19m
wd-discovery-converter 1/1 1 1 5h57m
wd-discovery-crawler 1/1 1 1 5h57m
wd-discovery-entity-suggestion 1/1 1 1 5h19m
wd-discovery-entity-training 1/1 1 1 5h19m
wd-discovery-gateway 1/1 1 1 4h11m
wd-discovery-glimpse-builder 1/1 1 1 5h17m
wd-discovery-glimpse-query 1/1 1 1 5h19m
wd-discovery-haywire 1/1 1 1 5h14m
wd-discovery-hdp-rm 1/1 1 1 5h15m
wd-discovery-ingestion-api 1/1 1 1 5h15m
wd-discovery-inlet 1/1 1 1 5h57m
wd-discovery-management 1/1 1 1 5h15m
wd-discovery-minerapp 1/1 1 1 4h18m
wd-discovery-orchestrator 1/1 1 1 5h57m
wd-discovery-outlet 1/1 1 1 5h57m
wd-discovery-po-box 1/1 1 1 5h57m
wd-discovery-project-data-prep-agent 1/1 1 1 5h14m
wd-discovery-ranker-master 1/1 1 1 5h14m
wd-discovery-ranker-monitor-agent 1/1 1 1 5h14m
wd-discovery-ranker-rest 0/1 1 0 5h14m
wd-discovery-rapi 1/1 1 1 5h15m
wd-discovery-serve-ranker 1/1 1 1 5h14m
wd-discovery-stateless-api-model-runtime 1/1 1 1 5h57m
wd-discovery-stateless-api-rest-proxy 1/1 1 1 5h57m
wd-discovery-support 0/0 0 0 5h15m
wd-discovery-tooling 1/1 1 1 4h18m
wd-discovery-training-agents 1/1 1 1 5h14m
wd-discovery-training-crud 1/1 1 1 5h14m
wd-discovery-training-rest 1/1 1 1 5h14m
wd-discovery-watson-gateway-gw-instance 1/1 1 1 4h8m
wd-discovery-wd-indexer 1/1 1 1 5h57m
wd-discovery-wksml 1/1 1 1 5h57m
wd-minio-discovery-kes 1/1 1 1 5h52m
Resolving The Problem
Patch the image with the latest one for IBM CPD 4.6.6:
oc patch wd wd --type=merge --patch='{"spec": {"wire": {"rankerRest": {"image": {"tag":"20240223-012927-645-b14d8a8","digest":"sha256:89e2c3efffbb06eb1605fb6b3b550ca7e0a41bc88f9e0a0d78c5727a54ff9635"}}}}}'
Know issue: DT381121 - Maximo Applicaation Suite Assist - WD ranker rest pod unable to start. Known issue with wd-discovery-ranker-rest in CPD 4.6.6
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB59","label":"Sustainability Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSRHPA","label":"IBM Maximo Application Suite"},"ARM Category":[{"code":"a8m3p000000F83HAAS","label":"Maximo Application Suite-\u003EMAS Applications-\u003EAssist"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.10.0;8.11.0"}]
Was this topic helpful?
Document Information
Modified date:
07 May 2024
UID
ibm17149820