IBM Sterling Order Management - Performance Guide

• Validate API input before calling API, ensure required or filtrable attributes are passed (avoid open ended SQL. 
• Limit the input size, and use optimal API output template to limit unnecessary data reads.
• Restrict output by setting the MaximumRecords in the inputs to any list API calls; use pagination.
• Tune servlet.token.absolute.timeout properties to prevent YFS_USER_ ACTIVITY locking under heavy load. Read more →
• Use appropriate SelectMethod method NO_LOCK, NO_WAIT, WAIT, and timeout properties QueryTimeout="3" TimeoutLockedUpdates="Y" for getter API’s
• Database Timeout properties for agent, integration, and UI: yfs.agentserver.queryTimeout, yfs.ui.queryTimeout
• Keep transaction boundary small when using update/modify API, this will ensure DB object (row) is locked for minimum duration.
• Use appropriate connect and read timeout for external calls, preferably less than 5 seconds, and make use of connection pool (cached/persistent connection, keep-alive).
• Remove always on DEBUG or SystemOut statements
• Eliminate frequent SELECT by enabling entity cache, at the same time make sure to redundant cache (always miss or frequently evicted)
  • Avoid using current timestamp value as part of query predicate (this makes caching redundant due to unique value)
• Avoid the use of multiApi to process bulk transaction synchronously, instead use asynchronous requests (via JMS to drop message on queue and process using integration server)

• Apply recommended JMS performance properties (See Integration > JMS Performance section)
• Review order and shipment monitors for redundancy, review and remove obsolete monitor rules.
• Have clear understanding of order ON_CHANGE or ON_SUCCESS events, make sure they are not excessive, and also validate for recursion.  
  • Excessively deep or infinite recursion can lead to java.lang.StackOverflowError exception and will halt JVM.
• Avoid reprocessing of order once condition evaluates to false. Read more yfs.yfs.monitor.stopprocessing.ifcondition.eval.false=Y
• Tune next task queue interval of "Process order hold type" agent from 15 minutes to the customized value yfs.omp.holdtype.reprocess.interval.delayminutes
• Have dedicated schedule order server to process backorders using OrderFilter= N|B agent criteria flag.
• Separate out the processing of orders by one of the attributes (ex. Large order, etc) with workload separation feature. Read more → 
• Apply and Tune OMoC default HOTSku and OLA configuration. Read more → 
• Enable Capacity cache and tune node locking properties based on business use case. Read more →
• Apply sourcing optimization (reduce DG size)
• When using YFSGetAvailabilityCorrectionsForItemListUE, make sure output of the UE excludes the items with ZERO supply quantity before passing the result to OOB API.
• Apply solver/sourcing interrupt properties to prevent runaway transactions within schedule agent. 
• If capacity is enabled, then make sure to double check the calendar setting (store hours, etc.) for peak.
• Control/Throttle use of createInventoryActivityList API when using capacity filled event.

• Apply recommended configuration around API performance (see above)
• Run purges prior to peak to ensure transaction tables such as YFS_ORDER_RELEASE_STATUS, YFS_ORDER_HEADER, YFS_ORDER_LINE, YFS_SHIPMENT, etc are lightweight to ensure optimal performance of the getter APIs.
• Cache critical configuration data using entity cache: YFS_REGION, YFS_REGION_DETAIL, YFS_ATTR_ALLOWED_VALUES, YFS_ATTR_ALLOWED_VAL_LOCALE
• Add the following indices to enhance the performance of the Batch Pick. Read more → 
  • Index on the STORE_BATCH_KEY column of the YFS_SHIPMENT_LINE table.
  • Index on the SHIPNODE_KEY and INCLUDED_IN_BATCH columns of the YFS_SHIPMENT table.
• Set the property yfs.applyChildContainerQueryOptimization=Y in DB properties to optimize the query on the shipment container table while fetching child containers.
• Execute closeManifest API asynchronously.
• Set polling interval of Store and Call Center dashboard widgets to avoid redundant calls.
• Execute GetStoreBatchList with optimum values for MaxNumberOfShipments, NoOfShipmentLinesForNewBatch
• Purge YFS_INBOX table, identify and address root cause of the exception, keep exception to minimal

• Use optimal API and output template (ATP vs. Promising API), query what’s needed:
  • Promising APIs such as findInventory would evaluate procurements, optimizations (such as Cost, Date, etc.), Shipment transfers, constraints (such as Ship Complete, Ship Single Node, etc.).
  • For Inventory check ATP APIs such as getATP, getAvailableInventory or getAvailablityCache can be used, these APIs will avoid promising optimizations.
  • Implement short lived availability cache for product browsing pages. 
• Use optimal sourcing & scheduling configuration:
  • Adjust Lead Times days according to the business needed; this can significantly reduce unnecessary compute. 
  • Implement solver and interrupt properties depending on the workload. See Common Performance Properties section.
• The sourcing should be designed in a way that nodes which are likely to be selected as final solution are only supplied from sourcing to availability and scheduling.
  • Having a smaller number of nodes in sourcing sequence by using region-based sourcing, proximity (i.e. with x miles), multiple sequence optimization can yield better performance.
  • All nodes can be last in the sequence to ensure order is fulfilled.
  • Smart sourcing be used with GIV-RTAM with node level monitoring to filter out nodes based on availability cache. 
  • Integrate with Sterling Intelligent Promising, it combines inventory and capacity visibility with sophisticated fulfillment decisioning to help retailers maximize inventory productivity, make reliable and accurate order promises, and optimize fulfillment decisions at scale
• If reservation node can be considered as final ship node, then it should be passed on order line. This avoids schedule order to consider sourcing again.
• Enable API interrupt properties to avoid runaway transactions
• When calling findInventory consider using MaximumRecords attribute, this takes positive integer used to determine the maximum number of possible solutions to find. Using MaximumRecords="1" can help improve the performance. By default 5 solutions are generated.
• Run Inventory purge
• Ensure performance testing reflects expected production usage such as inventory and capacity availability, distribution of orders between SFS, BOPIS, DC, etc.

• Promising API Interrupt (This can be used for AppServer wherein costly findInventory can be terminated):
  • yfs.IntruptAfterMinutesMode
  • yfs.IntruptAfterMinutes
  • yfs.IntruptModeForReadWrite
• Solver Interrupt:
  • yfs.yfs.solver.WarningOrExitOnIntrupt
  • yfs.yfs.solver.IntruptAfterMinutes
  • yfs.yfs.solver.IntruptOnlyForRead
• Optimization:
  • yfs.yfs.solver.MaxChoiceFailures
• Aggregate reservation calls to IV, this improves performance of reserveAvailableInventory API yfs.UseAggregatedReservationsForIV property to "Y".

• To disable node capacity do not adjust capacity to infinite.
  • Use changeResourcePool API to disable capacity for specific resource pool, or DISABLE_NODE_CAPACITY_FOR_ENT. 
  • If capacity is set to 0, the capacity consumption record is deleted from the YFS_RES_POOL_CAPCTY_CONSMPTN table. Use capacity purge to delete the 0 capacity records
• Use Capacity Cache; The time-triggered agent pre calculates capacity and pre-populates YFS_CAPACITY_AVAILABILITY table
  • Over allocation can be prevented using node locking yfs.nodecapacity.lock, yfs.nodecapacity.threshold properties.

• Review MessageBufferPutTime relative to ExecuteMessageCreated statistic from YFS_STATISTICS_DETAIL table for any slowness
• Use non-persistent queues for internal agent queues
• User persistent queues for external integration or integration server processes.
• Avoid using message Selector, instead have dedicated internal/external queues
• Avoid longer transaction to prevent MQRC_BACKED_OUT error message
• Optimize output template to prevent MQRC_MSG_TOO_BIG_FOR_Q error while posting a message
  • ​​Use message compression when handling large inbound or outbound message. Read more → 
    • Note: IBM Sterling Order Management on Cloud allows 4 MB message size. 
• Cache JMS bindings file (Specially if you serving the file from PV with slow I/O)
  • yfs.jms.sender.alwayslookupqueue.disabled=true
• Enable JMS Session Pool
  • yfs.yfs.jms.session.disable.pooling=N
• Enable multi-threaded PUT’s; Can help improve performance of RTAM server. Read More →
  • yfs.yfs.jms.sender.multiThreaded=true
• Use anonymous reuse (requires JMS Session pooling to be enabled)
  • yfs.jms.sender.anonymous.reuse=true
• Enable JMS connection retries. Read more → 
  • Retry Interval (milliseconds) 100 ms
  • Number of Retries – at least 3 retries.
• Enable agent bulk sender properties to POST message in batches.
  • yfs.agent.bulk.sender.enabled=Y|true
  • yfs.agent.bulk.sender.batch.size=5000

• Use connection pool (cached/persistent connection, keep-alive) with appropriate connect and read timeouts.
• Cache authentication token for reuse, regenerate upon expiry, or 401, 403 status codes. Read more → 
• Adhere to best practice when invoking Inventory Visibility APIs. Read more for updated best practices →
• Use 100 item-node per payload when invoking APIs for multiple lines.
• Implement polling process to retrieve failed events. Read more →
• Space out the sync supply and snapshot calls, check with all stakeholders for ad-hoc execution or special requests during peak.
  • Avoid redundant calls to generate snapshot
• Avoid redundant Network availability recomputes, consider following situations:
  • Recompute Network Availability API recomputes availability for existing DG.
  • Update DG API will recompute availability for newly created or modified DGs but not for existing DG.
  • Bulk update to turning on/off existing nodes. 
    • Note: This requirement to turn of fulfillment by type can be achieved by frontend feature flag. 
• Make sure to review release notes and API documents.
  • Timely refactor the logic to avoid running into unforeseen risks of using deprecated or discontinued APIs. Read more →
  • Upgrade to V2 APIs for IBM Sterling Intelligent Promising. Read more →

• Enabling property yfs.yfs.app.identifyconnection=Y to identify the source of DB query / connection.
• Enable and optimize entity cache.
• DB2: Enable stmt_conc (LITERALS)*
• Oracle: Avoid full table scan with ConsiderOracleDateTimeAsTimeStamp attribute when using Oracle database. Read more →
• Monitor the frequently invalidated table caches and disable them if needed
• Avoid Blank queries, one of the common use case when non optimal API input is passed in. Ensure APIs are invoked with key filtering attributes.
• Make sure SQLs aren’t formed with unique values at runtime, it impacts the cache reusability.
• Enable application performance features and properties required for concurrent workload to avoid DB contentions (HOTSku, Capacity, etc.)
• Watch for YFS_PERSON_INFO query; apply index if needed. Read more →
  • Check for duplicate records in YFS_PERSON_INFO table. 
• Disable unnecessary transaction (entity) audits (Order Audits, General Audits, etc.)
• Disable resource intensive database maintenance during peak period (such as REORGs & RUNSTAT)
• Tune / Avoid ad-hoc queries used for reporting purpose, if possible, use standby or replica instances to query.
• Monitor database for long queriers and queries in lock-wait, and transaction logs usage.

• Maintaining healthy database can prevent disruption in production.
• Reduce the IBM Sterling Order Management database size with entity level compression and enhanced purges. More details →
• Accumulation of transactional data over long periods of time (and failure to purge as possible), may degraded query performance. Ensure all necessary purges are running to maintain healthy & lightweight database, which in-turn minimizes performance issues.

                serverProfiles:
                - name: "profile-name"
                  resources:
                    limits:
                      cpu: millicores 
                      memory: bytes
                    requests:
                      cpu: millicores
                      memory:    bytes
                ...
                - name: agents-huge
                  profile: ProfileHuge
                  property:
                    customerOverrides: AgentProperties
                    envVars: EnvironmentVariables
                    jvmArgs: BaseJVMArgs
                  replicaCount: 1
                  agentServer: 
                    names:[ScheduleLargeServer,ReleaseLargeOrderServer]

            appServer:
              ingress:
                contextRoots: [smcfs, sbc, sma, wsc, isf]

Recommendations:

• Spawning additional (untuned) instances of agent to try and improve throughput can lead to exhaustion of resource allocation available, my stress the system, and it can have cascading impact to other transactions. 
• Do not solely rely on pod autoscaler, make sure to performance test for expected workload and scale replica's accordingly.

To best position for success on the OMS platform, it is important to understand how your application handles various scenarios known to challenges performance or stability. Testing in pre-production with data/workloads representative of production enables ability identify and address issues without impact to production business and operations. Performance testing is an art, but a mandatory one! It is imperative to vet out issues in advance on pre-production load testing, rather than wait for it to surface as a business-critical production issue!

• Projected peak volumes – Ensure business and IT are in sync on expected peak loads to ensure planned tests are accurate.
• Representative Combination Tests – Assemble components to reflect real time DATA, scenarios and run in parallel to ensure adherence with NFR; Stage data for various components and run them under full load (ie. Create + Schedule + Release+ Create Shipment + Confirm Shipment + Inventory Snapshot (IV) )
  • Ensure inventory picture (supply), node capacity (resource pool), distribution groups, and nodes setup reflect production state. 
  • Do not use same PI data, as it can lead to unexpected contention. In production PI data is unique. 
• Agent and integration servers – ensure asynchronous batch processing components are tested in isolation and in combination with broader workload; ensure to tune agents and integration server (profile, threads) to meet expected peak SLAs/NFRs on throughput
• Test Failure Scenarios – validate resiliency of overall system and operations, ensuring graceful recovery if front-end channel (web, mobile, Call Center, Store, EDI, JMS), backend OMS, or external integration endpoints fail. Include ‘kill switches’ in any components that can be disabled to avoid magnifying an isolated issue into system wide one, especially for any synchronous calls.
• Confirm Peak days and Hours - Share any specific key dates or max burst times with IBM Support, including code freezes, flash sales.

Note: Refer to Knowledge Center for detailed Tuning and performance guidance.

• Availability Check (Ping, Scripted Browser, API Test)

• Golden signals (Throughput, Latency, Error Rate, Saturation)
• Agent/Integration Server JVM Health (GC & Heap Utilization) 
• Application Server JVM Health (GC, Heap, Default Executor Threads)
• Container workload (Pod) (CPU throttling, availability of desired number of Pods, frequent restarts, etc.)
• Application Statistics: YFS_STATISTICS_DETAIL

• Queries in Lock-Wait
• Long Running Query
• Transaction Log
• Tablespace
• HADR log replay (replication lag monitoring)

• Queue depth
• Message delay or equivalent queueing/dequeuing rate for all JMS queues.

• Local, NFS Disk Utilization
• VM (host) not responding
• CPU, Memory, Disk Usage
• CPU Steal
• Open Sockets (ulimit)

• Up/Down stream latency
• Number of Queued Requests
• Average Queue Time
• Active Session
• Bytes Received/Sent Per Second