MustGather for IBM Order Management Software Certified Containers: Performance Issues

• If the Application Server is frozen or unresponsive, provide 3-4 sets of Thread dumps collected at intervals of 20 to 30 secs.
  • Identify the pod where slowness is observed
    Exec into the pod by using -

    oc exec -it name-ibm-oms-ent-prod-appserver-om-app-5bb8bb74c6-p5tww /bin/bash
    

    In the pod, run ps aux command to identify the process ID

    Sample Output

    bash-4.4$ ps aux
    USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    default        1  5.7 28.0 7458356 2233580 ?     SLsl Sep03 173:25 /opt/ibm/java/jre/bin/java -javaagent:/opt/ibm/wlp/bin/tools/ws-javaagent.jar -Djava.awt.headless=true -Djdk.attach.allowAttachSelf=true -Dvendor=websphere -
    default      858  0.0  0.0  12024  2548 pts/0    Ss+  Sep04   0:00 /bin/bash
    default     1624  0.0  0.0  12024  3184 pts/1    Ss+  14:41   0:00 /bin/bash
    default     1631  0.5  0.0  12024  3300 pts/2    Ss   14:44   0:00 /bin/bash
    default     1637  0.0  0.0  44632  3372 pts/2    R+   14:44   0:00 ps aux

    
    - Now run kill -3 <pid>
    - The javacore.date.time.id.txt will be generated under the log directory folder (specified in values.yaml file).

• Provide 2 set of logs -
  • Apply VERBOSE tracing on the identified Application or API 
  • Apply SQLDEBUG tracing on the identified Application or API 
  • Reference - Video that illustrates how to put components on trace
  • Note:  VERBOSE trace is not suitable for the production environment. If you are only able to reproduce the issue on production, then consider single user test with UserTracing enabled instead.
• For slowness or OOM issues, please share any generated verbose garbage collection logs and heap dumps.
• If the issue is identified to be due to Database Locking, upload the details requested here.
• Restart the server to mitigate the issue.

      params:
      - -Xhealthcenter:level:headless
      - -Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=/opt/ibm/wlp/output/defaultServer
      - -Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=15
      - -Dcom.ibm.diagnostics.healthcenter.data.profiling=off
      - -Dcom.ibm.java.diagnostics.healthcenter.allocation.threshold.low=10000000
      - -Dcom.ibm.java.diagnostics.healthcenter.stack.trace.depth=20
      - -Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=0

• If you are able to replicate slowness or OOM issues intermittently,
  • Enable verbose garbage collection and upload logs to the case. Steps to enable verboseGC logs are documented here.
  • Enable heap dumps on user events and upload the generated heap dumps to case. Steps to enable heap dumps are documented here. Please refer the IBM Documentation link which has more details regarding the same.

jvmArgs: "-Xms512m -Xmx1024m -Xhealthcenter:level:headless -Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=/shared/agents/$(OM_POD_NAME)/hcd 
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=15 -Dcom.ibm.diagnostics.healthcenter.data.profiling=off -Dcom.ibm.java.diagnostics.healthcenter.allocation.threshold.low=10000000 -Dcom.ibm.java.diagnostics.healthcenter.stack.trace.depth=20 -Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=0"

select statistics_detail_key, start_time_stamp, end_time_stamp,
server_name, server_id, hostname, service_name, service_type, context_name, statistic_name, statistic_value from yfs_statistics_detail where statistics_detail_key like '2021082514%'

select statistics_detail_key, to_char(start_time_stamp, 'YYYY-MM-DD HH24:mi:SS'), to_char(end_time_stamp, 'YYYY-MM-DD HH24:mi:SS'),
server_name, server_id, hostname, service_name, service_type, context_name, statistic_name, statistic_value
from yfs_statistics_detail
where statistics_detail_key like '2021082514%'
order by statistics_detail_key

 - -verbose:gc
 - -Xverbosegclog:/shared/logs/$(OM_POD_NAME)/gclogs/%Y%m%d.%H%M%S.%pid.txt
 - -Xgcpolicy:gencon

jvmArgs: "-Xms512m -Xmx1024m -verbose:gc -Xverbosegclog:/shared/agents/$(OM_POD_NAME)/gclogs/%Y%m%d.%H%M%S.%pid.txt --Xgcpolicy:gencon"