Data import
In Maximo® Manage, you can import data by using object structures, external systems, file systems, interface tables, REST APIs, or third-party tools.
Data import methods in Maximo Manage
- Application import by using object structures
- Supported file formats are XML, JSON, and flat files. You use the Maximo Manage user interface for this type of data import, for example, the Assets application or the Work orders application. It is a synchronous process. Do not use application import for large amounts of data due to possible user interface session timeouts. The time that is needed to import data depends on the data type and the amount of data that is being imported. For example, simple objects, such as Chart of Accounts, can be faster than more complex objects, such as work orders. Typically, use application import for less than 5000 records.
- External systems-based import
- Supported file formats are XML, JSON, and flat files. Use the External Systems application in the Maximo Manage user interface for this type of data import. It is an asynchronous process. The process of writing to a messaging queue is synchronous, while processing from a messaging queue to a Maximo business object is asynchronous. A file with 10000 records is written as 10000 messages in the messaging system. This type of data import can have timeouts both in the user interface and during the process of writing to messaging queues.
- File system-based import
- Supported file formats are XML, JSON, and flat files. This type of data import uses a file-based storage system, that is, a persistent volume claim (PVC), or an object-based system, such as the S3 storage system. Cron tasks are set up to import large amounts of data.
- Interface table-based import
- Supported file formats are flat on a database table, which is an interface table. You use interface table cron tasks for this type of data import. A common way to enter data into the interface table is to use SQL.
- REST API-based import
- Supported file formats are XML, JSON, and flat files. The REST client must be multi-threaded for better throughput. Use bulk APIs for best performance. For more information on bulk APIs, see Bulk operations. Because bulk import is a synchronous process, the import size must be tuned to avoid HTTP timeouts.
- Third-party tools
- You can use third-party tools, such as MXLoader, to import data.
The following sections focus on the more commonly used ways to import data into Maximo Manage.
File system-based import for bulk data
File system-based import, being an asynchronous process, is ideal for importing large amounts of data. Normally, one folder is used by one cron task. However, by using the file system-based import, it is possible to split the data import into multiple folders, each being used by a cron task. Then, writing to the inbound messaging queue is done through parallel processing.
Eventually, flat files are converted to XML. Using XML or JSON files is a faster process because XML or JSON files are directly imported without any conversion.
Object-based storage is slightly slower in processing data to a messaging queue than a file system-based storage. For each write action to the messaging queue, the Maximo Manage cron job must update the object storage with the information that a record is delivered to the messaging queue. File system-based storage generally provides faster access than object-based storage, which can impact the performance for importing bulk data. The loading to messaging queue performance is faster in a file-based system. The loading of data from the messaging queue to the Maximo Manage database is the same in both file-based and object-based systems.
The file extract option is available for file system-based import. For example, you want to load a file with 10000 records, and 100 records have errors during the data import. The file extract option can extract these failed 100 records into a single file. It is then possible to fix them and import them again. Without file extract, you must fix each of these erroneous records individually in the Message Reprocessing application of Maximo Manage. However, file extract requires more resources because the process must track the count of messages that are processed successfully and the ones that display an error. Use continuous queues that are processed by using message-driven beans (MDBs). MDBs process messages in parallel from the messaging queue, so the dataset must support parallel processing.
The message tracking option tracks a message when it goes from the file system or object-based system to the messaging queue and then to the Maximo Manage database. Message tracking tends to slow down the processing because it creates multiple entries for each record in the tracking table.
For more information on file system-based import, see Importing file-based data.
Interface table-based import for bulk data
Interface table-based import uses a database table instead of a file for importing data. Use interface tables if you want to use SQL for handling data. The interface table column names match the flat file column headers. The interface table data, which is similar to the flat file data, must be converted to XML before it is processed. The conversion happens in the interface table cron task component.
To process the data from the interface table to the messaging queue in multiple threads, you can split the data set in the interface table into multiple external systems. Then each thread can be assigned to an external system. The database schema contains N+1 tables, where N is the number of enterprise services or inbound interfaces that you want to import by using this mechanism. The +1 is the index or queue table that holds the reference or record identifiers and sequence of the records in the interface tables, which is N. The index or queue table is the mxin_inter_trans table, which contains the record ID, external system name, and the enterprise service name. The default table is the data table, for example, the mxin_asset_ifacetable table. The system identifies the data table based on the enterprise service name from the index or queue table. By using the external system name, you can distribute your records into each external system. Multiple threads are used, one for each external system to load the data faster into the messaging middleware, which is the JMS or Kafka queue.
For more information about interface tables, see Interface tables.
REST API-based import for bulk data
You can use REST APIs to import data from an external system or application into Maximo Manage. For more information on REST-API based import, see Data import and export with REST API.