Suggestions for Improving CPYF Performance

Resolving The Problem

Following are suggestions to improve performance when running many CPYF commands with logical files over each physical file:

1.	Break the CPYF command up into two segments by removing the logical file members, running the CPYF command, and adding the logical file members.
2.	Give as much memory and resources possible to the job. Set up two memory pools, one for CPYF commands and one for logical file builds. Having separate pools avoids main storage contention. The pool for CPYF must be about 400K per thread. The rest of the memory should be in the logical file pool. Start with the smaller files first so logical file builds can be started.
3.	Do the physical file loads with HLL (RPG) programs rather than using the CPYF command with *MAP *DROP. The RPG programs can move the data more quickly than a CPYF with *MAP *DROP because it can bind the variables to proper MI instructions at compile time. CPYF must do it at run time. To block the I/O of the RPG programs, do an OVRDBF SEQONLY(*YES n) where n is: CISC: 32K divided by the recl of the file, RISC: 128K divided by the recl of the file. Do not define the file specifications as keyed access.
4.	Create logical files with the most key fields first. Create the logical files in order of those with most key fields first through those with least key fields. This ensures access path sharing whenever possible.
5.	Do not multi-thread the logical file build. Running one logical file build per available CPU on the system is recommended. Multi-thread of logical files that are over different physical files is recommended, but logical files over the same physical file should be single threaded.
6.	Use expert cache.
7.	Get started with more than one physical file load. Because there are no logical files to build until physical files are copied, start out with two physical file copies running in the logical file pool and one in the physical file pool. This helps maximize throughput until your logical file builds gets going.

At R530 and above, the following also helps with CPYF performance:

1.	Remove the keys on the To file to allow the copy to run faster and create an index over the key with the key fields.
2.	Use SQL to copy the file over.
3.	Use FMTOPT(*NOCHK) on the CPYF to remove checking the data. Keep in mind this may cause bad data to be put into the file.

Also, if using CPYF to move data between two supposedly identical files, it is important that the record format level identifiers are the same to obtain the quickest performance. If the record format level identifiers are not the same, the CPYF command goes through a longer code path that does field/column level checking for each record. This could impact the time the CPYF command requires to complete the function when such checking is not required (for example, 17 minutes versus 3 hours 50 minutes on a 20 million record/row copy).

Likewise, the format level identifiers should be different if the files have different attributes. Otherwise, field/column level checking is not performed when it should be, which results in improper data conversions or none at all. The FMTOPT(*NOCHK) parameter can be specified to avoid field/column level checking (when it might not be needed) regardless of the value of the record format level identifiers. However, specific attributes of the data (such as null values) are lost when the FMTOPT(*NOCHK) parameter is specified.