How to manage build dependencies

Direct build dependencies

z/OS programs often have references to other source files that are needed by the compiler to build the program. Examples of such references are the COBOL COPY, the PL/I %INCLUDE, the C #include, SQL INCLUDE, and the Assembler macro reference. These references are known as direct build dependencies and are important to identify the source code because:

1. Precompilers and compilers need the latest version of the dependency source files to build the program correctly. This might require copying the dependency files from z/OS file system (zFS) directories to partitioned data sets (PDS) before building the program.
2. They are needed to support building applications incrementally where only programs or their build dependencies that have changed since the last build are rebuilt.

Indirect build dependencies

Indirect build dependencies define relationships between the program source and the outputs from previous build processes. While indirect dependencies are not considered during dependency resolution, they are a factor in impact analysis.

Impact analysis refers to the process of identifying which programs in an application are impacted by code changes to copybooks or include files recursively. If a source file is changed and rebuilt, impact analysis can determine other sources that need to be rebuilt as a result of that output changing. An incremental build uses impact analysis to build only programs that are out of date either because the program has been modified or a copybook or include file that it uses has been modified since the last time the application was built.

There are two types of indirect build dependencies:

zUnit Test build dependencies

IBM Developer for z/OS (IDz) v14.2.1 introduced the IBM z/OS Dynamic Test Runner for zUnit, which greatly increases the zUnit test capabilities that are provided by IDz including the capability to record and playback z/OS applications. This allows for the easy capture of test data and the ability to run tests in a batch environment with no middleware configured.

DBB provides the capability to run these tests as part of a CI/CD pipeline by providing a zUnit Test Configuration Dependency Scanner, which will automatically create dependency relationships between the z/OS source program, the test program, and the playback data file. These dependency relationships can then be used by DBB build scripts to automatically copy zUnit test case files from the local Git repository to partitioned data sets to run zUnit test cases as part of a pipeline build. These dependency relationships also allow DBB to run zUnit test cases incrementally when the z/OS program changes or when the related test program or the playback file is modified.

1. Scanning source file dependencies

Dependency collection begins with scanning source code files for build dependencies. DBB provides a multi-language dependency scanner that can be used to find dependencies for Assembler, C/C++, COBOL, and PL/I source files. The DependencyScanner class is located in the com.ibm.dbb.dependency package.

// Scan a single file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)

// Scan an archived file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/copybook.tar.gz" // tar or tar.gz file
def logicalFileList = new DependencyScanner().scanArchive(file, sourceDir) // returns logical file list

// Scan files listed in an external file
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
   def logicalFile = scanner.scan(file, sourceDir)
   logicalFiles.add(logicalFile)
}

As seen in the example, the DependencyScanner scan method takes two String arguments:

• file - This is the source file's path relative to the sourceDir of the file to scan. It is used as the key of the logical file when it is stored in the DBB repository.
• sourceDir - This is the absolute path of the directory on the local file system that contains the source files. If the source directory is a local Git repository, it is the path of the directory that contains the .git folder. It is used to locate the file in the local file system in order to scan its contents.

The result of running the scan method is a LogicalFile that contains the dependency information of the scanned file. For more information about the LogicalFile class, see Resolving logical build dependencies to physical files.

The result of running the scanArchive method is a LogicalFile list that contains the dependency information of all of the files contained within the archive.

2. Scanning zUnit configuration file dependencies

The information in this section is focused primarily on scanning IDz zUnit configuration files to create and store DBB dependency relationships between the z/OS source file, the zUnit test case program, and the zUnit playback file. For a complete step-by-step guide for integrating zUnit tests into a CI/CD pipeline using DBB see Integrating Unit Tests into an open and modern CI/CD pipeline.

A zUnit Test Case is composed of the following items:

• A configuration file: An XML file that contains the relationship of the test case with the source program, the test program, and the playback file. When it is generated by IDz, it has a .bzucfg file extension.
• A test program.
• A zUnit playback file to drive the process. It is not always required.
• A JSON file with the contents of the test data editor, which is not used as part of a CI/CD build.

The DBB ZUnitConfigScanner class extends the AbstractDependencyScanner class and provides all of the same scan methods as does the DBB source file DependencyScanner class outlined above.

Note: The Files scanned by the zUnitConfigScanner must be valid IBM z/OS Dynamic Test Runner for zUnit configuration files. Invalid configuration files will result in a warning message generated to the console and a logical file with no dependency relationships created.

The ZUnitConfigScanner class is located in the com.ibm.dbb.dependency package.

// Scan a zUnit configuration file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/testcfg/epscmort.bzucfg"
def logicalFile = new ZUnitConfigScanner().scan(file, sourceDir)

The result of running the scan method is a LogicalFile that contains the dependency information of the zUnit configuration file. Unlike the z/OS source DependencyScanner, which returns a LogicalFile with any number of logical dependencies depending on the number of copy statements or include statements in a source file, the logical file generated from the ZUnitConfigScanner contains only three logical dependencies:

3. Creating a collection to store source file dependency data

To use the build dependency information collected by the DependencyScanner and ZUnitConfigScanner for dependency resolution and impact analysis, the resulting logical files of all the scanned source files (programs, dependency files and zUnit configuration files) must be stored in the DBB repository database as part of a dependency Collection. A collection is a repository container for logical files. The scope of a collection is determined by the user. For example, a collection can contain all the logical files of a Git branch, from multiple Git repositories, or with other files added. This way, the logical files in a collection can use the scanned file's relative path from the sourceDir as a unique identifier in the collection. Collections themselves can have any name but it is recommended to use the name of the Git branch of the source files being scanned.

Example of a logical file stored in a DBB Repository Collection

Collections are repository artifacts. So, communication with an active DBB server is required to create and modify them. For more information about initializing the RepositoryClient utility class, see Repository client.

// Create a collection to store scanned dependency data
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists("MortgageApplication.master"))
   client.createCollection("MortgageApplication.master") 

Collection names must be unique in the DBB repository database. An error occurs when you try to create a collection that already exists. A good practice is to first check whether the collection with that name exists before you create it.

Unlike the build result, a dependency collection is a simple repository object that contains only a list of logical files. As such, there is no dedicated interface for it. All collection APIs are in the RepositoryClient utility class.

// Create a collection to store scanned dependency data
def collectionName = "MortgageApplication.master"
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists(collectionName))
   client.createCollection(collectionName) 

// Add or update a logical file to a collection
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)
client.saveLogicalFile(collectionName, logicalFile)

// Retrieve a logical file from a collection
logicalFile = client.getLogicalFile(collectionName, "MortgageApplication/cobol/epsnbrvl.cbl")

// Delete a logical file from a collection
client.deleteLogicalFile(collectionName, "MortgageApplication/cobol/epsnbrvl.cbl")

// Add a list of logical files to a collection
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
   def logicalFile = scanner.scan(file, sourceDir)
   logicalFiles.add(logicalFile)
}
client.saveLogicalFiles(collectionName, logicalFiles)

The RepositoryClient also provides two methods for searching a collection for logical dependencies:

• getAllLogicalFiles(collectionName, logicalName) - finds all the logical files in a collection that match the logical searched name. Use it to include the dependency source files to build the program correctly.
• getAllLogicalFiles(collectionName, LogicalDependency) - finds all the logical files in a collection that match the set fields of a logical dependency. Use it for impact analysis during incremental builds.

// Find all logical files that have the logical name (lname) = EPSNBRVL i.e. MortgageApplication/cobol/epsnbrvl.cbl
def lfiles = client.getAllLogicalFiles(collectionName, "EPSNBRVL")

// Find all logical files that have a copy dependency on EPSMTCOM i.e. contain a COPY EPSMTCOM statement
def lname = "EPSMTCOM"
def category = "COPY"
def library = null  // null fields are ignored in the logical file search
def logicalDependency = new LogicalDependency(lname, library, category)
lfiles = client.getAllLogicalFiles(collectionName, logicalDependency)

// Find all logical files that have a dependency reference that resides in DD library MYLIB
logicalDependency = new LogicalDependency(null, "MYLIB", null)
lfiles = client.getAllLogicalFiles(collectionName, logicalDependency)

4. Scanning static link dependencies

Link dependency collection begins with program object scanning. DBB provides a LinkEditScanner that scans program objects to determine the relationship between build process outputs and source files. This relationship is captured with link dependencies.

Note: Link-edit scanning is limited to program objects in a partitioned data set extended (PDSE). While the link-edit scanning works against a load module in a PDS, the information needed for impact analysis and dependency resolution is not available. If the final result of a link-edit must be in a PDS, you are recommended to link-edit twice, once to a PDSE and again to a PDS, to obtain the necessary LINK dependencies from the PDSE. The PDSE can be discarded at the end of the build.

The LinkEditScanner class is located in the com.ibm.dbb.dependency package.

// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)

As seen in the example, the LinkEditScanner scan method takes three String arguments:

• file - The source file's path relative to the sourceDir of the file to scan. It is used as the key of the logical file when it is stored in the DBB repository.
• loadPDS - The fully qualified PDS containing the program object.
• member - The program object member name.

The result of running the scan method is a LogicalFile that contains the link dependency information of the scanned program object. The LogicalFile class is covered more in depth in the Resolving logical build dependencies to physical files section.

// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def scanner = new LinkEditScanner()
scanner.setExcludeFilter("DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*");
def logicalFile = scanner.scan(file, loadPDS, member)

dbb.LinkEditScanner.excludeFilter = DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*

5. Creating a collection to store link-edit dependencies

In order for the impact analysis to use the indirect dependency information collected by the link-edit scanner, the resulting logical files of all the scanned program objects must be stored in the DBB repository database in a link dependency collection that is separate from the source file dependency data collection. A link dependency collection is a repository container for logical files that relate to the outputs from the link-edit scanner. Collections themselves can have any name, but it is recommended to use a similar but different name as the related dependency collection.

Note: The link dependency collection must be a separate collection from the source file dependency data collection. If placed into the same collection, the logical files and link dependencies might be overwritten when the source is rescanned. Both collections must be added to the ImpactResolver during impact analysis.

Collections are repository artifacts. So, communication with an active DBB server is required to create and modify them. For more information about initializing the RepositoryClient utility class, see Repository client.

// Create a collection to store indirect dependency data
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists("MortgageApplication.master.outputs"))
   client.createCollection("MortgageApplication.master.outputs") 

Collection names must be unique in the DBB repository database. An error occurs when you try to create a collection that already exists. A good practice is to first check whether the collection with that name exists before you create it.

// Create a collection to store indirect dependency data
def collectionName = "MortgageApplication.master.outputs"
def client = new RepositoryClient().url("dbb.server.company.com:9443/dbb").userId("usr1").password("usr1_pw")
if (!client.collectionExists(collectionName))
   client.createCollection(collectionName) 

// Add or update a logical file to an indirect dependency collection
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)
client.saveLogicalFile(collectionName, logicalFile)

6. Resolving logical build dependencies to local physical files

As mentioned earlier, one of the primary reasons to collect and store application dependency data is to make sure that the build uses the latest versions of the build dependencies when compiling programs or test cases. However, all DBB has scanned and collected so far are the logical dependencies of a program or test case. To locate those dependencies so that DBB can copy the latest versions from local Git repositories in zFS to data sets for inclusion during a build, you need to resolve their locations on zFS. zFS is a z/OS UNIX® System Services (z/OS UNIX) file system that can be used in addition to the hierarchical file system (HFS).

Note: DBB v1.1.2 introduces a simpler API for resolving logical build dependencies to local physical files as an alternative to the DependencyResolver class originally provided by DBB v1.0. This section discusses using the new SearchPathDependencyResolver class for dependency resolution. For information on how to use the original DependencyResolver class to resolve logical dependencies, see the DBB v1.0.x documentation for resolving logical dependencies to physical files.

You can use the SearchPathDependencyResolver class to resolve the logical dependencies in a program's logical file to physical files on the local zFS file system. It searches for each dependency by using a formatted search path string. More information on creating the searchPath string is described below.

// Create a dependency search path which searches '/u/build/repo' and all its sub-directories for files with a '.cpy' file extension 
String searchPath = "search:/u/build/repo?path=**/*.cpy"

// Create a search path dependency resolver to resolve dependencies using this searchPath 
def resolver = new SearchPathDependencyResolver(searchPath)

// Locate all the copybooks used by program epscmort.cbl on the zFS file system
List<PhysicalDependency> dependencies = resolver.resolveDependencies("cobol/epscmort.cbl", "/u/build/repo");

The result from running the SearchPathDependencyResolver.resolveDependencies method is a list of physical dependencies List<PhysicalDependency>. The PhysicalDependency class extends the LogicalDependency class with information needed to locate the dependency file in the local zFS file system.

Notes:

• The SearchPathDependencyResolver.resolveDependencies method is a recursive search. If PGM1 references CPYBOOKA and CPYBOOKA references CPYBOOKB, then both COPYBOOKA and CPYBOOKB will be listed in the physical dependencies for PGM1.

• Some dependencies like "DFHAID" found in programs that execute CICS commands or like "SQLCA" found in programs that execute SQL queries are returned as unresolved dependencies. The reason is that they are not usually located in a user's SCM and therefore are not scanned and added to a DBB collection. However, it is not an issue because the reason for collecting the dependency information is to copy the files from zFS to data sets before the build. These dependencies already reside in data sets, so nothing needs to be copied.

Now that the logical dependencies of a program have been resolved to physical dependencies, they can be copied from their locations zFS to data sets for inclusion in compilation.

// Iterate through a list of physical dependencies to copy them to a data set
def physicalDependencies = resolver.resolveDependencies(searchPath)
physicalDependencies.each { physDep ->
    if (physDep.isResolved()) {
       File file = new File("${physDep.getSourceDir()}/${physDep.getFile()}")
       new CopyToPDS().file(file).dataset("USR1.BUILD.COPYBOOK").member(physDep.getLname).execute()
    }
}

As the CopyToPDS command class has built-in support for copying a list of physical dependencies, the above example can be much simplified.

def physicalDependencies = resolver.resolveDependencies(searchPath)
new CopyToPDS().dependencies(physicalDependencies).dataset("USR1.BUILD.COPYBOOK").execute()

search:[[<library>:<category>:<lname>]]<sourceDir>|<archiveFile>[?path=<globPattern>;...]
  |                   |                           |                       |
-----         -----------------            ---------------         ---------------
scheme        dependency filter              search root             search path

// Resolve the dependencies
def props = BuildProperties.getInstance()
def physicalDependencies = logicalFile.resolveDependencies(props.dependencySearch)

# full workspace or archive file search path - for unique file names
dependencySearch = search:/u/build/repo
dependencyArchiveSearch = search:/u/build/lib/copybooks.tar.gz

# full workspace or archive file search path for specific file extension
dependencySearch = search:/u/build/repo?path=**/*.cpy
dependencyArchiveSearch = search:/u/build/lib/copybooks.tar.gz?path=**/*.cpy

# targeted search with specific path
copybookSearch = search:/u/build/repo?path=MortgageApplication/copybook/*.cpy
copybookArchiveSearch = search:/u/build/lib/copybooks.tar.gz?path=copybook/*.cpy

# search paths using dependency filters
syslibFilterSearch = search:[SYSLIB]/u/build/repo?path=${application}/copybook/*.cpy
syslibCopylibFilterSearch = search:[SYSLIB,COPYLIB]/u/build/repo?path=${application}/copybook/*.cpy
linkCategoryFilterSearch = search:[:LINK]/u/build/repo?path=${application}/cobol;${application}/link
lnameWildcardFilterSearch = search:[::EPSN*,EPSC*]/u/build/repo?path=${application}/copybook/*.cpy

# dependency exclude list filter search
lnameExcludes = ^DFH*,^CEE*,^XCH*,^XWS*,^SQLCA,^DSN*
excludeDependencyFilterSearch = search:[SYSLIB::${lnameExcludes}]/u/build/repo?path=${application}/copybook/

7. Identifying programs impacted by changed copybooks, include files, and statically linked programs

An incremental build uses impact analysis to only build the programs that are out of date or have dependency references (such as copybooks, include files, or macros) that are out of date.

Note: DBB v1.1.2 introduces a simpler API for determining which programs are impacted by dependency source file changes and need to be rebuilt as an alternative to the ImpactResolver class originally provided by DBB v1.0. This section discusses using the new API for impact analysis and resolution. For information on how to use the original ImpactResolver class to resolve impacted files, see the DBB v1.0.x documentation.

DBB provides the SearchPathImpactFinder class to run an impact analysis on changed copybooks and include files. Whereas the SearchPathDependencyResolver starts with a program and finds all of the copybooks and include files that the program needs to build, the SearchPathImpactFinder class goes the other way. It starts with a changed copybook or include file and finds all of the programs, copybooks and include files that reference it. To do so, it queries the collections in the build repository to see what logical files have a dependency reference to the changed copybook or include file being searched. More information on creating the searchPath string is described above.

All of the searches are run against DBB repository collections so communication with an active DBB server is required. For more information about initializing the RepositoryClient utility class, see Repository client. Also note that the more up-to-date the logical files in the collections to be searched, the more accurate the impact analysis is.

// Start by creating a search path which accepts changed files from '/u/build/repo' and all its sub-directories for files with a '.cpy' file extension 
String searchPath = "search:/u/build/repo?path=**/*.cpy"

// Create a list of collections to search
List<String> collections = new ArrayList<String>()
collection.add("MortgageApplication");

// Instantiate a SearchPathImpactFinder to find programs impacted by changed copybooks 
def finder = new SearchPathImpactFinder(searchPath, collections, repositoryClient)

// Find all files impacted by changed copybook 
List<ImpactFile> findImpacts = finder.findImpactedFiles("copy/epsmtcom.cpy", "/u/build/repo")

The searchPath argument is optional and can be a Java or Groovy null value if all the dependency logical names (lname) are unique within a DBB collection. This is often the case in migrated z/OS source file applications. However, if it is not the case, then the searchPath defined in the SearchPath discussed above can be reused.

The result of running the finder.findImpactedFiles method is a list of impacted files, that is, List<ImpactFile>. An ImpactFile represents a program or other source file that has a build dependency on the changed file.

• lname - The logical name of the program. For example, the lname of epsmtinp.cbl is EPSMTINP.
• file - The program's or other source file's path relative to the sourceDir used to scan the file. It is the same value as the file attribute in a logical file.
• language - The language of the source file.
• collection - The name of the collection where the impacted program or source file's logical file is.

Note: The SearchPathImpactFinder.findImpactedFiles method is a recursive search. The list of the impacted files returned by the method contains all source files that are directly and indirectly impacted by the changed file. For example, when an impact resolution is done on CPYBOOKB, if CPYBOOKB is referenced by CPYBOOKA and if CPYBOOKA is referenced by PRG1, then both CPYBOOKA and PRG1 will be in the list of impacted files for CPYBOOKB.

String impactSearchPath = "search:/u/build/repo?path=**/*.cpy;**/*.bms";

String collectionName = "MortgageApplication.master";
String outputsCollectionName = "MortgageApplication.master.outputs";
List<String> collections = new ArrayList<String>();
collections.add(collectionName);
collections.add(outputsCollectionName);

// create the collections list
String collectionName = "MortgageApplication.master";
String outputsCollectionName = "MortgageApplication.master.outputs";
List<String> collections = new ArrayList<String>();
collections.add(collectionName);
collections.add(outputsCollectionName);

// create and concatenate the search paths
String copybookSearch = "search:[SYSLIB]/u/build/repo?path=MortgageApplication/copybook;MortgageApplication/bms";
String programSearch = "search:[:LINK]/u/build/repo?path=MortgageApplication/cobol";
String impactSearch "$copybookSearch$programSearch;

// create SearchPathImpactFinder and find inpacts
SearchPathImpactFinder finder = new SearchPathImpactFinder(impactSearch, collections, repositoryClient)
List<ImpactFile> resolveImpacts = finder.findImpactedFiles("MortgageApplication/cobol/epscmort.cbl", "/u/build/repo");