Troubleshooting
Problem
Analytics data sporadically missing from UI or the storage and ingestion pods restarting themselves.
Symptom
If you are observing problems with missing analytics data in the API Connect Manager and Portal UIs, or seeing analytics storage pods restarting themselves, it might be that your analytics deployment's Java heap size allocation needs to increase from the default values.
Diagnosing The Problem
Check the logs of the analytics ingestion and storage pods.
Storage
The storage pod names are slightly different depending upon the version of API Connect you are using:
For v10.0.5.0 and later, you should always check the logs for the pods with a name similar to 'analytics-storage-x' (where x is a number).
For versions prior to v10.0.5.0, the pod names will depend on the analytics storage type. When storage type is 'shared', the pod name will be similar to 'analytics-storage-shared-x'. When storage type is 'unique', the pod name will be similar to 'analytics-storage-data-x'.
You are looking for output in the logs that refer to the time being spent on garbage collection.
For example:
[analyt-storage-shared-0] [gc][12765060] overhead, spent [ 2.2 s] collecting in the last [3.1 s]
Garbage collection is a normal process, so it is expected to see these messages in the storage pod logs. However, when these messages become very frequent, and indicate the process is spending a large proportion of its time on garbage collection then the performance of analytics will be impacted.
For example. if you see these messages every few seconds, then there is likely insufficient heap memory size. However, if you see this message every 10 minutes or so, and it says spent [ 220 ms ] collecting in the last [1 s] then the message is not usually a concern.
Ingestion
If there is a problem with the heap memory size for ingestion, you might see the following error in the ingestion pod(s):
java.lang.OutOfMemoryError: Java heap space
Resolving The Problem
API Connect 10.0.5 introduced deployment profiles, which is the recommended method to change the heap size. It's recommended to change the deployment profile for the Analytics subsystem instead of editing the heap. The advantage of switching deployment profiles is it changes the resource requests and limits for all pods, not just the storage pods.
It is possible to increase the Java heap size, but care must be taken not to increase the heap excessively. First, try increasing by 50% and then see whether the problems are resolved.
IMPORTANT: Ensure your Kubernetes environment or VMs have sufficient RAM available before you attempt these steps.
Storage
NOTE: An alternative to increasing the heap size for the storage pod is to reduce the retention period, which can reduce the load on the storage pods.
Kubernetes, OpenShift or CP4I deployment
For v10.0.5.2 and later only: Switch to a deployment profile that will automatically increase the heap size for both storage and ingestion microservices. See the following documentation for details:
If you still need to override the heap size settings, follow these steps:
1. Take a backup of your analytics CR with:
kubectl get a7s analytics -o yaml > <MEANINGFUL_NAME>.yaml
2. Edit the analytics CR:
kubectl edit a7s analytics
and paste the appropriate YAML block, indented as part of the 'spec:' section:
v10.0.5.0 and later
template:
- name: storage
containers:
- name: storage
env:
- name: OPENSEARCH_JAVA_OPTS
value: "-Xms6g -Xmx6g" # this is the OpenSearch heap size
resources:
requests:
memory: 12Gi # this is the guaranteed memory for the pod(s); it should be larger than the heap size
limits:
memory: 12Gi # this specifies the maximum memory the pod(s) can use; it should be double the heap size
Versions prior to v10.0.5.0
template:
- name: storage-data # for 'shared' storage type, this microservice is called 'storage-shared'
containers:
- name: storage-data # for 'shared' storage type, this container is called 'storage-shared'
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g" # this is the Elasticsearch heap size
resources:
requests:
memory: 12Gi # this is the guaranteed memory for the pod(s); it should be larger than the heap size
limits:
memory: 12Gi # this specifies the maximum memory the pod(s) can use; it should be double the heap size
You must update the values for the 'name' parameters in this example if you are using the 'shared' storage type to 'storage-shared'.
You must be careful of indentation and whitespace; the file will not be saved if there is any problem with the syntax.
The values in the example are suggested minimum values. If you are experiencing problems, you could start by increasing the heap size to 9g ("-Xms9g -Xmx9g"), and the requests and limits to 18Gi (ensure your Kubernetes environment has sufficient RAM available).
Save and exit. The storage pod(s) should automatically restart.
3. Verify the change is applied with:
kubectl describe statefulset <statefulset_name>
(Run 'kubectl get sts' first to identify the statefulset name for the storage pods)
You should see the memory and the ES_JAVA_OPTS (or OPENSEARCH_JAVA_OPTS - v10.0.5.0+) settings at their new values. For example:
storage-shared:
...
Limits:
...
memory: 4Gi
Requests:
...
memory: 3Gi
...
ES_JAVA_OPTS: -Xms2g -Xmx2g
4. Check storage pod logs to verify that the frequency and time spent doing garbage collection has reduced.
OVA deployment
You can edit the CR as described for non-OVA deployments, but this does not persist through an upgrade. Instead, set these values in the extra-values file. See:
Add the following to the extra-values file:
v10.0.5.0 and later
spec:
template:
- name: storage
containers:
- name: storage
env:
- name: OPENSEARCH_JAVA_OPTS
value: "-Xms6g -Xmx6g"
resources:
requests:
memory: 12Gi
limits:
memory: 12Gi
Versions prior to v10.0.5.0
spec:
template:
- name: storage-shared
containers:
- name: storage-shared
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g"
resources:
requests:
memory: 12Gi
limits:
memory: 12Gi
Then, run:
apicup subsys install analytics
Note: if you've not previously setup the analytics extra-values file, you need to run
apicup subsys set analyt extra-values-file path/analytics-extra-values.yaml
before you run apicup subsys install.
Ingestion
Note: These steps are only required if logs from the ingestion pod indicate that there is insufficient heap memory (see Diagnosing The Problem).
The process for increasing the heap size for the ingestion microservice is similar to what has been described for the storage microservice, but instead the settings are applied to the ingestion microservice and container.
kubectl edit a7s
Immediately under the spec: section, add (changing highlighted values as necessary):
template:
- name: ingestion
containers:
- name: ingestion
env:
- name: LS_JAVA_OPTS
value: "-Xms2g -Xmx2g"
resources:
requests:
memory: 4Gi
limits:
memory: 4Gi
Note: If you have already added a template section to increase storage heap size, do NOT add a 2nd 'template' section. Instead, add the '- name: ingestion' section as a peer (same indent) of the '-name: storage-data' section, inside the same 'template' section. For example:
template:
- name: storage-data
containers:
- name: storage-data
env:
- name: ES_JAVA_OPTS
value: "-Xms6g -Xmx6g"
resources:
requests:
memory: 12Gi
limits:
memory: 12Gi
- name: ingestion
containers:
- name: ingestion
env:
- name: LS_JAVA_OPTS
value: "-Xms2g -Xmx2g"
resources:
requests:
memory: 4Gi
limits:
memory: 4Gi
For OVA edit the extra-values file in the same way as for the storage microservice.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000CeBlAAK","label":"API Connect-\u003EManagement and Monitoring (MM)-\u003EAnalytics"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.0.0;and future releases"}]
Was this topic helpful?
Document Information
Modified date:
17 July 2024
UID
ibm16549832