History Mass Sync agent
OSI_HISTORY_MASS_SYNC
agent to migrate all the existing history
orders to Order Search and stage the orders to
OSI_AWAITING_ARCHIVE
table for archival.
Run the OSI_HISTORY_MASS_SYNC
agent immediately after enabling the built-in
integration with Order Search and Order Archival.
As Order Search is the single source of truth for all the orders that are present in Sterling Order Management System Software and orders that are archived to Order Archival, it is important to run this agent as one time activity to migrate all the existing history orders to Order Search before archiving or making the orders eligible for archival.
- SaveIndexName: You can create indexes with specific names in Order Search to migrate history orders of a particular year or for a year range. To migrate history orders, it is mandatory to provide the Save Index Name.
- SyncFromYear and SyncToYear: An order for which the order date is between the range of year specified by using this criteria are picked up for processing. For history order migration, it is mandatory to specify the SyncFrom and SyncTo year.
- You can run multiple instances of
OSI_HISTORY_MASS_SYNC
agent with different criteria parameters, but make sure that they do not define criteria with overlappingSyncFromYear
andSyncToYear
criteria parameters. - The
OSI_HISTORY_MASS_SYNC
agent is an enterprise agent. - Before running the Order Archival agent to archive orders, migrate all the existing history orders to Order Search and back up the corresponding Order Search indexes.
- The
OSI_HISTORY_MASS_SYNC
agent processes the history order records, which are not already staged for archival in theOSI_AWAITNG_ARHIVE
table. - Each
executeJob
runs on a ‘batch’. Each such batch corresponds to a range of entity records, for example order records from a particular colony. Furthermore, batches from different colonies are interspersed. Therefore, you can configure multiple agent threads spanning across multiple agent servers for faster indexing. - Since the agent runs for several records of the entity tables in each
executeJob
call, multiple failure scenarios can exist:- If critical problems are encountered that affect all index attempts, for example if the index
server is unreachable, the
executeJob
of the agent fails with an exception. - If individual documents fail, failure is reported though the Exception Notification framework. Such records are reattempted in the subsequent trigger of this agent.
- If critical problems are encountered that affect all index attempts, for example if the index
server is unreachable, the
- History Mass Sync agent looks at records from YFS_ORDER_HEADER_H and OSI_AWAITING_ARCHIVE tables
as part of
getJob
to prepare theexecuteJob
messages. It is recommended to closely monitor the performance of database queries that are executed from the History Mass Sync agent and add additional indices to get optimum throughput. Based on the testing that IBM carried with specific data setup (15 million orders in YFS_ORDER_HEADER_H table), following additional indices, helped improve the query performance.- An index on column ENTITY_KEY of OSI_AWAITING_ARCHIVE table
- An index on columns ORDER_DATE and DRAFT_ORDER_FLAG of YFS_ORDER_HEADER_H table
- Before running the agent, ensure that you set the
ORDERSERVICE_USE_INDEX_NAME_IN_REQUEST
property in Order Search. Once the agent run is completed and all the existing history orders are migrated, reset the property. - Draft orders are not supported for search and archival. Hence, this agent does not process history draft orders.
Attributes
The following table describes the History Mass Sync agent attributes:
Attribute | Value |
---|---|
Base Transaction ID | OSI_HISTORY_MASS_SYNC |
Base Process Type | GENERAL |
Criteria Parameters
The following table describes the History Mass Sync agent criteria parameters:
Parameter Name | Description |
---|---|
Action | Required. This parameter triggers the transaction. The default value is "Get". |
BatchSize | Number of records to process in a single
executeJob . An execute message is created for this batch of records. The default
value is 500. |
NumBatchesToBuffer | Number of batches (execute messages) to create in one iteration
of getJobs . The default value is 1000. |
Number of Records To Buffer | Maximum number of records to retrieve from database in one database query. The default value is 50000. |
EnterpriseCode | The enterprise code of the enterprise for which the History Mass Sync agent must be run. |
IndexName | Name of the index. For example, Order. |
ColonyId | Indicates the colony ID. |
Collect Pending Jobs | The History Mass Sync agent does not provide the pending jobs count for monitoring. |
SyncFromYear | The year from which orders are picked up for processing based on the Order Date. |
SyncToYear | Year up to which orders are picked up for processing based on the OrderDate. |
SaveIndexName | Name of the index created in Order Search that is used by History Mass Sync agent to migrate the history orders. |
History Mass Sync agent is designed to create job messages from a huge volume of data. For
example, in case of orders, there can be about 10 to 20 million orders in the order history table.
To create jobs, the records are fetched from an ordered list, to ensure processing of every record
and to avoid duplicate attempts. Fetching smaller set of data in an ordered fashion from million
records leads to numerous highly expensive database queries. Therefore, the total time that is
required to create jobs for such million records is longer. However, the time required to fetch 1000
or 50000 records does not differ much and therefore you must set a higher value for the
Number of Records To Buffer
parameter. This helps to pull more data from the
database in one attempt, which leads to lesser database queries and results in creating the required
messages quickly.
The default parameter values create 1000 jobs to process 500,000 records. The number of records
to process is deduced from NumBatchesToBuffer
multiplied by
BatchSize
. However, as the default Number of Records To Buffer
is
50000, it invokes only 10 database queries to create 1000 messages. You can double both
NumBatchesToBuffer
and Number of Records To Buffer
to double the
number of records to process from one iteration. However, it is recommended to double the value of
Number of Records To Buffer
, to get the same job done in reduced time.
Do not increase the BatchSize
to a higher value as it increases the HTTP payload
to Order Service.
Number of Records To Buffer
must be a multiple ofBatchSize
.NumBatchesToBuffer
must be a multiple of the batches created from theNumber of Records To Buffer
, which isNumber of Records To Buffer
divided byBatchSize
.
Events Raised
None
Statistics Tracked
None
Pending Job Count
None