IBM Support

FileNet document upload using ACCE with more than 370MB can cause CPE pod crash and ACCE to return error

Troubleshooting


Problem

Creation of a document using the FileNet Administration Console for Content Platform Engine (ACCE) with content sized more than about 370MB or uploads of multiple documents with a total content size of more than about 370MB can cause the Content Platform Engine (CPE) pod to be evicted due to insufficient memory or out of disk space for ephemeral storage.

Symptom

Creation of large document using ACCE can present an error like this indicating the connection to the CPE was lost:
Error web application might not be running

Cause

ACCE stages the uploaded content in a temporary location in the CPE container as a part of the content element upload and creation process. If the content is large or multiple files are uploaded within a short amount of time, the location in the CPE container where the file is staged may become full. Since this temporary location is ephemeral storage, as a result of it becoming full, the Kubernetes platform evicts (terminates and recreates) the CPE pod causing the connection from ACCE to be lost.

Diagnosing The Problem

Access the cluster and examine the deployment for the CPE that ACCE was connected to. Determine if a CPE pod was restarted during the time period when ACCE was disconnected by examining the events for the CPE deployment recorded by Kubernetes.

Resolving The Problem

 The location where uploaded files are staged is the temporary location used by the CPE JVM and controlled by the JVM option java.io.tmpdir. In a default deployment, the temporary location is created as ephemeral storage in the container which has limited space available.  
To resolve this issue, change the staging temporary location for the JVM from ephemeral storage to the /opt/ibm/textext mounted volume which is mapped to a persistent volume and can be made larger.
Edit the custom resource YAML (CR) used to create the FileNet Content Manager deployment and make two changes. 
1) Set the size of the persistent volume claim for the text extraction location to be sufficient to support the amount of content uploaded using ACCE within a fifteen minute time period.  If the PVC already exists but is too small, use the Kubernetes command line or administration console to increase the size. By default the name of the PVC is "cpe-textextstore".
Add the following to the ecm_configuration.cpe section of the CR specifying a size that will accommodate multiple uploads of the size of content expected for documents to be created using ACCE:

     datavolume:
        existing_pvc_for_cpe_textextstore: 
          name: "cpe-textextstore"
          size: 10Gi
2) Add the following to the ecm_configuration.cpe section of the CR to move the JAVA temporary location to the text extraction location:
 
      cpe_production_setting:
        jvm_customize_options: -Djava.io.tmpdir=/opt/ibm/textext
 Apply the modified CR and wait for the Operator reconcile to apply the changes.
After the CPE pods restart, retry the document creation operation using ACCE.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSNVNV","label":"FileNet Content Manager"},"ARM Category":[{"code":"a8m0z000000bpDjAAI","label":"ACCE"},{"code":"a8m0z0000001h6UAAQ","label":"Container"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.5.10;5.5.11"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSGLW6","label":"IBM Content Foundation"},"ARM Category":[{"code":"a8m0z000000bpDjAAI","label":"ACCE"},{"code":"a8m0z0000001h6UAAQ","label":"Container"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.5.10;5.5.11"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBYVB","label":"IBM Cloud Pak for Business Automation"},"ARM Category":[{"code":"a8m0z0000001iUBAAY","label":"Use-\u003EFNCM App Usage"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"22.0.2;23.0.1"}]

Document Information

Modified date:
11 September 2023

UID

ibm17031218