IBM Support

Install Process Mining with Asset Management, IAF Core Operator crash

Troubleshooting


Problem

Case

IAF Core Operator is crashing when IPM instance is added to cluster on top of Asset Management Install

Symptom

Customer installed using two scenario after installing cp4a correctly:
Customer using this to install Process Mining:  https://www.ibm.com/docs/en/cloud-paks/1.0?topic=platform-how-install-process-mining-operator:
Customer using this to install Asset Manager: https://www.ibm.com/docs/en/cloud-paks/1.0?topic=assets-installing-automation
Scenario 1: My starting point is having a fully installed CP4BA 21.0.3.7.
Then if I add just ProcessMining, it seems OK and IAF Core is not crashing. On top of CP4BA and IPM I add either Asset Repository or RPA and IAF Core starts crashing and installation of these subsequent components doesn't work.
Scenario 2: My starting point is having a fully installed CP4BA 21.0.3.7. Then if I add Asset Repository and RPA and everything is OK and IAF Core is not crashing. On top of CP4BA with Asset Repository and RPA, I add IPM and when I add its instance CR, IAF Core starts crashing. Trying these different combinations and sequences leads me to conclusion that the error is cause by a combination of multiple things - IAF Core, IPM and other components. The case is that the problem occurs only when IPM is part of the deployment. Additionally I did another test. If I scale down IPM Operator, IAF Core keeps crashing. If I wait and grab a certain moment when IAF Core operator is spinning up again before crashing again I can change the Cartridge called processmining and change its spec.version from v1.1 to v1. This change makes IAF Core operator to stop crashing. But when I scale IPM Operator back, it replaces the Cartridge with v1.1 and IAF Core starts to crash again.
When viewing all pods, customer saw IAF pod failed. In the logs, we can see in IAF core operator log:
E0413 21:21:09.197211 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

Cause

There is an inconsistency between the install instructions between process mining and asset management where asset management is using the :latest tag and process mining is pointing to a specific tag. The conflict was caused by: ibm-automation-foundation-core-catalog

Environment

Production - Asset Management and PM 1.12.0.4

Diagnosing The Problem

When checking issue, we first checked the pods on the environment, noticed that IAF core is failing and is doing a constant FailureLoop status even when deleting the pod.
We checked inside the pod, and it was throwing:
When viewing all pods, customer saw IAF pod failed. In the logs, we can see in IAF core operator log:
E0413 21:21:09.197211 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Investigating IAF operator instances in Workspace/Operator/, we can see there is cartridges created for CP4A, ProcessMining, and Asset Manager where only process mining cartridge is not in ready status.
Upon reading the Process Mining Cartridge  Instance throws error: Failed to create configmap cloud-pak/iaf-zen-tour-customization-extensions: resourceVersion should not be set on objects to be created
Noticed that these are created after instances are created for process mining and asset management, deleting the Process Mining Operator instance deletes the corresponding cartridge.

Resolving The Problem

For my instance, I checked and it is already using the latest PM instance at this moment 1.12.0.4.

I went to documentation page for Asset Manager and Process Mining and analyzed the specific dependencies for both and noticed that if blindly followed the installation, the problem will arise with conflict of dependencies. When installing, please check if the dependencies already exists. The resolution for going forward for prevention is to check if there are overlaps in the dependencies.

The fix for the environment would be to uninstall the Process Mining Instance/ Asset Management Instance/ IAF Core Operator and go to the official document to install the IAF Core Operator first from Process Mining documentation in dependencies, and then install Process Mining Instance and Asset Management Instance.

 install Process Mining:  https://www.ibm.com/docs/en/cloud-paks/1.0?topic=platform-how-install-process-mining-operator:
 install Asset Manager: https://www.ibm.com/docs/en/cloud-paks/1.0?topic=assets-installing-automation

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBYVB","label":"IBM Cloud Pak for Business Automation"},"ARM Category":[{"code":"a8m3p000000Go3RAAS","label":"Other-\u003EProcess Mining-\u003EPM Install\\Upgrade\\Setup"}],"ARM Case Number":"TS009037818","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
19 May 2022

UID

ibm16574441