MustGather for IBM Sterling Order Management: Performance Issues

1. Identify the server that is crashing and share details about the incident (that is, continuous vs intermittent) along with the timestamps of start/end time. 
2. Query the yfs_statistics_detail table for the server and share the data.
   • Db2:

     • server_name, server_id, hostname, service_name, service_type, context_name, statistic_name, statistic_value from yfs_statistics_detail where statistics_detail_key like '2021082514%'

   • Oracle:

     • select statistics_detail_key, to_char(start_time_stamp, 'YYYY-MM-DD HH24:mi:SS'), to_char(end_time_stamp, 'YYYY-MM-DD HH24:mi:SS'),
       server_name, server_id, hostname, service_name, service_type, context_name, statistic_name, statistic_value
       from yfs_statistics_detail
       where statistics_detail_key like '2021082514%'
       order by statistics_detail_key

3. If you know that the server is crashing due to java.lang.OutOfMemoryException, then share the following:
   1. Memory parameters (Xms, Xmx) for the server
   2. Customers on a OnPrem environment can share the heap memory dumps (refer to corresponding Step 1).

1. Share application logs from SST.
   • For more details refer to this page.

1. Share 3/4 sets of thread/heap memory dump from the affected server at 20-30 sec intervals.
   • You can issue kill -3 <PID> command to collect javacore and heap memory dumps.
     • This command does not "kill" the agent server process. It sends the Java process a signal 3 and creates a heapdump and a javacore.
     • Most JVMs' redirect thread dumps to system out logs. It is necessary to redirect system out of JVM process to some log file to obtain thread dumps (that is, java <complete command> > <LOG_FILE> 2>&1)
     • IBM Java redirects thread dumps to its own file. Look for an output file in the installation root directory with a name like javacore.date.time.id.txt
2. Share application logs from the affected server.
   • You can view agent/integration server logs at the following directory <runtime>/logs/sci**.log and <runtime>/logs/agentserver**.log.

1. Share the general time stamps of when this issue is observed.
   • For OnPrem environments, also share the DB maintenance schedule if it corresponds to the issue time frame.
2. If you are able to identify the source of the DB contention, query the yfs_statistics_detail table for the server and share the data.
   • Db2:

     • server_name, server_id, hostname, service_name, service_type, context_name, statistic_name, statistic_value from yfs_statistics_detail where statistics_detail_key like '2021082514%'

   • Oracle:

     • select statistics_detail_key, to_char(start_time_stamp, 'YYYY-MM-DD HH24:mi:SS'), to_char(end_time_stamp, 'YYYY-MM-DD HH24:mi:SS'),
       server_name, server_id, hostname, service_name, service_type, context_name, statistic_name, statistic_value
       from yfs_statistics_detail
       where statistics_detail_key like '2021082514%'
       order by statistics_detail_key

3. Ensure that the following property is set in customer.overrides: yfs.yfs.app.identifyconnection=Y.
   • This property helps in identifying DB connection-related details and is already enabled by default for SaaS environments.
   • This is a diagnostic property and should be used with caution in higher environments.
   • Read more about the property here.
4. Share SQLDebug logs from the affected server.
   • You have to enable SQL Debug tracing for the affected server from the System Management Console. For more details see this page.
5. For OnPrem environments on Oracle DB:
   • Share AWR report during incident time
   • Run the following queries 3- 4 times every 1 min apart to identify blocking locks (blocking_lock_sqls.txt).
     • These queries provide complete details on all JVMs contributing to the blocking locks.
6. For OnPrem environments on Db2:
   • Run the oms_db2collect_v2.sh script to gather Db2 configuration and performance data. The collection is lightweight and provides a high-level overview of the database performance. 
     • Copy oms_db2collect_v2.sh script to the Db2 server. Provide appropriate read, write, and execute permissions.
     • Run the script - Usage - ./oms_db2collect.sh <dbname>
     • Script generates db2collect.<timestamp>.zip in the folder where it is run

1. Have you tried calling the APIs directly through API tester? Share your observations on whether the slowness is observed only through the UI or through the API tester.
   • If you are able to identify which API is causing the slowness, you can enable SQLDebug trace on the API itself and share the logs along with a timestamp of when the issue is observed.
   • If you are unable to identify which API is causing the slowness, share the application server logs.
     • For SaaS environments, you can run the "Export application logs" process on SST for the server. For more details see this page.
     • For OnPrem environments, you can view agent/integration server logs at the following directory <runtime>/logs/sci**.log and <runtime>/logs/agentserver**.log.
2. Share details on whether the issue is reproducible across environments.
3. Share some high-level details on the customizations that have been done on this screen, if any.
4. Share which User Exits and Events were implemented as part of this flow.
5. Share the HAR logs from the browser.
   • The process for collecting HAR logs differs based on which browser is being used to access the site.
   • For more details see this page.

1. Share customer_overrides.properties file so that we can verify the caching as well as RMI properties configured on the environment.  
2. Share application logs along with timestamps of when the issue is observed.
3. For OnPrem environments on Oracle DB:
   • Share AWR report during incident time
   • Run the following queries 3- 4 times every 1 min apart to identify blocking locks (blocking_lock_sqls.txt).
     • These queries provide complete details on all JVMs contributing to the blocking locks.
4. For OnPrem environments on Db2, run the oms_db2collect_v2.sh script to gather Db2 configuration and performance data. The collection is lightweight and provides a high-level overview of the database performance. 
   • Copy oms_db2collect_v2.sh script to the Db2 server. Provide appropriate read, write, and execute permissions.
   • Run the script - Usage - ./oms_db2collect.sh <dbname>
   • Script generates db2collect.<timestamp>.zip in the folder where it is run
5. Query the yfs_heartbeat table for the service or server name and share the output.

Back to top