MustGather: Out of Memory errors with IBM API Connect v2018

Resolving The Problem

1. Ensure that CPU, Memory, and Disk  system requirements are met.
2. If deployed via OVA, provide the output of the following commands and all syslogs: 
   • `apic version --semver`
   • `apic logs`
   • `kubectl describe pods -n <namespace>`
   • All syslogs from var/log directory on each management subsystem VM (syslog, syslog.1, syslog.2.gz, etc)
   • Please upload a few sets of the dump files from the directory to which they were written. For example, var/db/java_dumps/ directory if the Cassandra pod experiences a java/lang/OutOfMemoryError: 
     JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" 
     JVMDUMP010I Heap dump written to /var/db/java_dumps/heapdump.20190112.013029.129.0006.phd
     JVMDUMP010I Java dump written to /var/db/java_dumps/javacore.20190112.013029.129.0007.txt
     JVMDUMP010I Snap dump written to /var/db/java_dumps/Snap.20190112.013029.129.0008.trc
   • • First instance (oldest timestamps)
           • heapdump*
           • javacore*
           • snap*
     • Last instance (newest timestamps)
           • heapdump*
           • javacore*
           • snap*
3. If deployed via Kubernetes, provide the output of the following commands:
   • `kubectl version`
   • `helm version`
   • Collect Helm release values for each APIConnect helm chart that is deployed with:  `helm get values $APIC_RELEASE --all`
   • `helm ls -a`
   • `kubectl get pods -a -n <namespace>`
   • `kubectl get endpoints -a -n <namespace>`
   • `kubectl get ingress -a -n <namespace>`
   • `apicup version --semver`
   • `kubectl -n kube-system logs -l component=kube-apiserver > kube-apiserver.out`
   • `kubectl -n kube-system logs -l component=kube-controller-manager > kube-controller-manager.out`
   • `kubectl -n kube-system logs -l component=kube-scheduler > kube-scheduler.out`
   • `kubectl -n kube-system get events > kube-system-events.out`
   • `kubectl get pvc -n <namespace>`
   • `kubectl describe pods -n <namespace>`
   • Copy the files created in java_dumps directory for the pod which experienced the error. For example  if the Cassandra pod experiences a java/lang/OutOfMemoryError gather the files via the following command:
     • `kubectl cp <namespace>/<cassandra-pod-0-name>:/var/db/java_dumps <path-to-local-system>`
   • Collect log files for all pods in a namespace with the attached script ( get_pod_logs.zip ) and upload resulting apic-logs.tgz. It will need slight modification prior to execution. Alternatively, a script can be created from the content found below

# $APIC_NAMESPACE is the namespace where APIConnect was deployed

# $LOG_DIR is a destination directory for the log files, any empty dir should do

for __pod in $(kubectl get pods -n $APIC_NAMESPACE --show-all -o name | cut -d'/' -f2); do

    for __container in $(kubectl get pod -n $APIC_NAMESPACE $__pod -o jsonpath="{.spec.containers[*].name}"); do

        kubectl logs -n $APIC_NAMESPACE $__pod -c $__container &> $LOG_DIR/${__pod}_${__container}.log

        kubectl logs --previous -n $APIC_NAMESPACE $__pod -c $__container &> $LOG_DIR/${__pod}_${__container}__previous.log

        [ $? -eq 0 ] || rm -f $LOG_DIR/${__pod}_${__container}__previous.log

    done

done

tar -C $LOG_DIR -cz -f apic-logs.tgz .

Submit the following output/files:

1. Output from all the commands in step 2 or 3 above, including heapdump, javacore & snap files needed to debug the cause of the OutOfMemoryError
2. apiconnect-up.yml file.
3. Collect apicup plan directory if available by zipping up the plan directory that was used for installation (in case sharing certificates is an issue, skip certs directories).