How to manage build dependencies

Direct build dependencies

z/OS programs often have references to other source files that are needed by the compiler to build the program. Examples of such references are the COBOL COPY, the PL/I %INCLUDE, the C #include, SQL INCLUDE, and the Assembler macro reference. These references are known as direct build dependencies and are important to identify the source code because:

1. Precompilers and compilers need the latest version of the dependency source files to build the program correctly. This might require copying the dependency files from z/OS file system (zFS) directories to partitioned data sets (PDS) before building the program.
2. They are needed to support building applications incrementally where only programs or their build dependencies that have changed since the last build are rebuilt.

Indirect build dependencies

Indirect build dependencies define relationships between the program source and the outputs from previous build processes. While indirect dependencies are not considered during dependency resolution, they are a factor in impact analysis.

Impact analysis refers to the process of identifying which programs in an application are impacted by code changes to copybooks or include files recursively. If a source file is changed and rebuilt, impact analysis can determine other sources that need to be rebuilt as a result of that output changing. An incremental build uses impact analysis to build only programs that are out of date either because the program has been modified or a copybook or include file that it uses has been modified since the last time the application was built.

There are two types of indirect build dependencies:

zUnit Test build dependencies

IBM Developer for z/OS (IDz) v14.2.1 introduced the IBM z/OS Dynamic Test Runner for zUnit, which greatly increases the zUnit test capabilities that are provided by IDz including the capability to record and playback z/OS applications. This allows for the easy capture of test data and the ability to run tests in a batch environment with no middleware configured.

DBB provides the capability to run these tests as part of a CI/CD pipeline by providing a zUnit Test Configuration Dependency Scanner, which will automatically create dependency relationships between the z/OS source program, the test program, and the playback data file. These dependency relationships can then be used by DBB build scripts to automatically copy zUnit test case files from the local Git repository to partitioned data sets to run zUnit test cases as part of a pipeline build. These dependency relationships also allow DBB to run zUnit test cases incrementally when the z/OS program changes or when the related test program or the playback file is modified.

1. Scanning source file dependencies

Dependency collection begins with scanning source code files for build dependencies. DBB provides a multi-language dependency scanner that can be used to find dependencies for Assembler, C/C++, COBOL, and PL/I source files. The DependencyScanner class is in the com.ibm.dbb.dependency package.

// Scan a single file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)

// Scan an archived file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/copybook.tar.gz" // tar or tar.gz file
def logicalFileList = new DependencyScanner().scanArchive(file, sourceDir) // returns logical file list

// Scan files listed in an external file
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
   def logicalFile = scanner.scan(file, sourceDir)
   logicalFiles.add(logicalFile)
}

As seen in the example, the DependencyScanner scan method takes two String arguments:

• file - This is the source file path relative to the sourceDir of the file to scan. It is used as the key of the logical file when it is stored in the DBB build metadata database.
• sourceDir - This is the absolute path of the directory on the local file system that contains the source files. If the source directory is a local Git repository, it is the path of the directory that contains the .git folder. It is used to locate the file in the local file system in order to scan its contents. If sourceDir is set to (String) null, then file is used as an absolute path.

The result of running the scan method is a LogicalFile that contains the dependency information of the scanned file. For more information about the LogicalFile class, see Resolving logical build dependencies to physical files.

The result of running the scanArchive method is a LogicalFile list that contains the dependency information of all of the files contained within the archive.

def logicalFile = new DependencyScanner().scanStream(file, inputStream)

2. Scanning zUnit configuration file dependencies

The information in this section is focused primarily on scanning IDz zUnit configuration files to create and store DBB dependency relationships between the z/OS source file, the zUnit test case program, and the zUnit playback file. For a complete step-by-step guide for integrating zUnit tests into a CI/CD pipeline using DBB see Integrating Unit Tests into an open and modern CI/CD pipeline.

A zUnit Test Case is composed of the following items:

• A configuration file: An XML file that contains the relationship of the test case with the source program, the test program, and the playback file. When it is generated by IDz, it has a .bzucfg file extension.
• A test program.
• A zUnit playback file to drive the process. It is not always required.
• A JSON file with the contents of the test data editor, which is not used as part of a CI/CD build.

The DBB ZUnitConfigScanner class extends the AbstractDependencyScanner class and provides all of the same scan methods as does the DBB source file DependencyScanner class outlined above.

Note: The Files scanned by the zUnitConfigScanner must be valid IBM z/OS Dynamic Test Runner for zUnit configuration files. Invalid configuration files will result in a warning message generated to the console and a logical file with no dependency relationships created.

The ZUnitConfigScanner class is located in the com.ibm.dbb.dependency package.

// Scan a zUnit configuration file
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/testcfg/epscmort.bzucfg"
def logicalFile = new ZUnitConfigScanner().scan(file, sourceDir)

The result of running the scan method is a LogicalFile that contains the dependency information of the zUnit configuration file. Unlike the z/OS source DependencyScanner, which returns a LogicalFile with any number of logical dependencies depending on the number of copy statements or include statements in a source file, the logical file generated from the ZUnitConfigScanner contains only three logical dependencies:

3. Creating a collection to store source file dependency data

To use the build dependency information collected by the DependencyScanner and ZUnitConfigScanner for dependency resolution and impact analysis, the resulting logical files of all the scanned source files (programs, dependency files and zUnit configuration files) must be stored in the DBB build metadata database as part of a dependency Collection. A collection is a database object that acts as a container for logical files. The scope of a collection is determined by the user. For example, a collection can contain all the logical files of a Git branch, from multiple Git repositories, or with other files added. This way, the logical files in a collection can use the scanned file's relative path from the sourceDir as a unique identifier in the collection. Collections themselves can have any name but it is recommended to use the name of the Git branch of the source files being scanned.

IBM Dependency Based Build 2.0.0    

LOGICAL FILE

DETAILS:

collection: MortgageApplication-main
lname: EPSMLIST
file: MortgageApplication/cobol/epsmlist.cbl
language: COB
cics: true
sql: false
dli: false
mq: false


LOGICAL DEPENDENCIES: 5

lname       library     category
--------    --------    --------
EPSNBRPM    SYSLIB      COPY
EPSMTCOM    SYSLIB      COPY
DFHAID      SYSLIB      COPY
EPSMLIS     SYSLIB      COPY
EPSMORTF    SYSLIB      COPY

// Create a build group to create the colletion to store scanned dependency data
def groupName = "MortgageApplication-main"
def collectionName = "sources"
def collection = null
def metadataStore = MetadataStoreFactory.createDb2MetadataStore(id, passwordFile, db2ConnectionProps)
if (!metadataStore.collectionExists(collectionName))
   collection = metadataStore.createCollection(collectionName) 

// Add or update a logical file to a collection
def sourceDir = "/u/build/repo"
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def logicalFile = new DependencyScanner().scan(file, sourceDir)
collection.addLogicalFile(logicalFile)

// Retrieve a logical file from a collection
logicalFile = collection.getLogicalFile("MortgageApplication/cobol/epsnbrvl.cbl")

// Delete a logical file from a collection
collection.deleteLogicalFile("MortgageApplication/cobol/epsnbrvl.cbl")

// Add a list of logical files to a collection
def filelist = new File("/u/build/buildList.txt") as List<String>
def logicalFiles = [] as List<LogicalFile>
def scanner = new DependencyScanner()
filelist.each { file ->
   def logicalFile = scanner.scan(file, sourceDir)
   logicalFiles.add(logicalFile)
}
collection.addLogicalFiles(logicalFiles)

// Find all logical files that have the logical name (lname) = EPSNBRVL i.e. MortgageApplication/cobol/epsnbrvl.cbl
def lfiles = collection.getLogicalFiles("EPSNBRVL")

// Find all logical files that have a copy dependency on EPSMTCOM i.e. contain a COPY EPSMTCOM statement
def lname = "EPSMTCOM"
def category = "COPY"
def library = null  // null fields are ignored in the logical file search
def logicalDependency = new LogicalDependency(lname, library, category)
lfiles = collection.getLogicalFiles(logicalDependency)

// Find all logical files that have a dependency reference that resides in DD library MYLIB
logicalDependency = new LogicalDependency(null, "MYLIB", null)
lfiles = collection.getLogicalFiles(logicalDependency)

4. Scanning static link dependencies

Link dependency collection begins with program object scanning. DBB provides a LinkEditScanner that scans program objects to determine the relationship between build process outputs and source files. This relationship is captured with link dependencies.

Note: Link-edit scanning is limited to program objects in a partitioned data set extended (PDSE). While the link-edit scanning works against a load module in a PDS, the information needed for impact analysis and dependency resolution is not available. If the final result of a link-edit must be in a PDS, you are recommended to link-edit twice, once to a PDSE and again to a PDS, to obtain the necessary LINK dependencies from the PDSE. The PDSE can be discarded at the end of the build.

The LinkEditScanner class is located in the com.ibm.dbb.dependency package.

// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)

As seen in the example, the LinkEditScanner scan method takes three String arguments:

• file - The source file path relative to the sourceDir of the file to scan. It is used as the key of the logical file when it is stored in the DBB metadata database.
• loadPDS - The fully qualified PDS that contains the program object.
• member - The program object member name.

The result of running the scan method is a LogicalFile that contains the link dependency information of the scanned program object. The LogicalFile class is covered more in depth in the Resolving logical build dependencies to physical files section.

// Scan a single program object
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def scanner = new LinkEditScanner()
scanner.setExcludeFilter("DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*");
def logicalFile = scanner.scan(file, loadPDS, member)

dbb.LinkEditScanner.excludeFilter = DFH.V3R2M0.CICS.SDFHLOAD.*, CEE.SCEELKED.*

5. Creating a collection to store link-edit dependencies

In order for the impact analysis to use the indirect dependency information collected by the link-edit scanner, the resulting logical files of all the scanned program objects must be stored in the DBB metadata database in a link dependency collection that is separate from the source file dependency data collection. A link dependency collection is a database object that is a container for logical files that relate to the outputs from the link-edit scanner. Collections themselves can have any name, but it is recommended to use a similar but different name as the related dependency collection.

Note: The link dependency collection must be a separate collection from the source file dependency data collection. If placed into the same collection, the logical files and link dependencies might be overwritten when the source is rescanned. Both collections must be added to the SearchPathImpactFinder during impact analysis.

As collections are database artifacts, an active metadata store connection is required to create and modify them. For more information about initializing the MetadataStore utility class, see Metadata store.

Collection names must be unique in the DBB metadata database. An error occurs when you try to create a collection that already exists. A good practice is to first check whether the collection with that name exists before you create it.

// Create a collection to store indirect dependency data
def collectionName = "MortgageApplication-main-outputs"
def collection = null
def metadataStore = MetadataStoreFactory.createDb2MetadataStore(id, passwordFile, db2ConnectionProps)
if (!metadataStore.collectionExists(collectionName))
   collection = metadataStore.createCollection(collectionName) 

// Add or update a logical file to an indirect dependency collection
def file = "MortgageApplication/cobol_cics_db2/epscmort.cbl"
def loadPDS = "${properties.hlq}.LOAD"
def member = "EPSCMORT"
def logicalFile = new LinkEditScanner().scan(file, loadPDS, member)
collection.addLogicalFile(logicalFile)

6. Resolving logical build dependencies to local physical files

As mentioned earlier, one of the primary reasons to collect and store application dependency data is to make sure that the build uses the latest versions of the build dependencies when compiling programs or test cases. However, all DBB has scanned and collected so far are the logical dependencies of a program or test case. To locate those dependencies so that DBB can copy the latest versions from local Git repositories to data sets for inclusion during a build, you need to resolve their locations on the z/OS UNIX® file system.

You can use the SearchPathDependencyResolver class to resolve the logical dependencies in a program's logical file to physical files on the local zFS file system. It searches for each dependency by using a formatted search path string. More information on creating the searchPath string is described below.

// Create a dependency search path which searches '/u/build/repo' and all its sub-directories for files with a '.cpy' file extension 
String searchPath = "search:/u/build/repo?path=**/*.cpy"

// Create a search path dependency resolver to resolve dependencies using this searchPath 
def resolver = new SearchPathDependencyResolver(searchPath)

// Locate all the copybooks used by program epscmort.cbl on the zFS file system
List<PhysicalDependency> dependencies = resolver.resolveDependencies("cobol/epscmort.cbl", "/u/build/repo");

The result from running the SearchPathDependencyResolver.resolveDependencies method is a list of physical dependencies List<PhysicalDependency>. The PhysicalDependency class extends the LogicalDependency class with information needed to locate the dependency file in the local zFS file system.

Notes:

• The SearchPathDependencyResolver.resolveDependencies method is a recursive search. If PGM1 references CPYBOOKA and CPYBOOKA references CPYBOOKB, then both COPYBOOKA and CPYBOOKB will be listed in the physical dependencies for PGM1.

• Some dependencies like "DFHAID" found in programs that execute CICS commands or like "SQLCA" found in programs that execute SQL queries are returned as unresolved dependencies. The reason is that they are not usually located in a user's SCM and therefore are not scanned and added to a DBB collection. However, it is not an issue because the reason for collecting the dependency information is to copy the files from zFS to data sets before the build. These dependencies already reside in data sets, so nothing needs to be copied.

Now that the logical dependencies of a program have been resolved to physical dependencies, they can be copied from their locations zFS to data sets for inclusion in compilation.

// Iterate through a list of physical dependencies to copy them to a data set
def physicalDependencies = resolver.resolveDependencies("cobol/epscmort.cbl", "/u/build/repo")
physicalDependencies.each { physDep ->
    if (physDep.isResolved()) {
       File file = new File("${physDep.getSourceDir()}/${physDep.getFile()}")
       new CopyToPDS().file(file).dataset("USR1.BUILD.COPYBOOK").member(physDep.getLname()).execute()
    }
}

As the CopyToPDS command class has built-in support for copying a list of physical dependencies, the above example can be much simplified.

def physicalDependencies = resolver.resolveDependencies("cobol/epscmort.cbl", "/u/build/repo")
new CopyToPDS().dependencies(physicalDependencies).dataset("USR1.BUILD.COPYBOOK").execute()

search:[[<library>:<category>:<lname>]]<sourceDir>|<archiveFile>[?path=<globPattern>;...]
  |                   |                           |                       |
-----         -----------------            ---------------         ---------------
scheme        dependency filter              search root             search path

// Resolve the dependencies
def props = BuildProperties.getInstance()
def physicalDependencies = logicalFile.resolveDependencies(props.dependencySearch)

# full workspace or archive file search path - for unique file names
dependencySearch = search:/u/build/repo
dependencyArchiveSearch = search:/u/build/lib/copybooks.tar.gz

# full workspace or archive file search path for specific file extension
dependencySearch = search:/u/build/repo?path=**/*.cpy
dependencyArchiveSearch = search:/u/build/lib/copybooks.tar.gz?path=**/*.cpy

# targeted search with specific path
copybookSearch = search:/u/build/repo?path=MortgageApplication/copybook/*.cpy
copybookArchiveSearch = search:/u/build/lib/copybooks.tar.gz?path=copybook/*.cpy

# search paths using dependency filters
syslibFilterSearch = search:[SYSLIB]/u/build/repo?path=${application}/copybook/*.cpy
syslibCopylibFilterSearch = search:[SYSLIB,COPYLIB]/u/build/repo?path=${application}/copybook/*.cpy
linkCategoryFilterSearch = search:[:LINK]/u/build/repo?path=${application}/cobol;${application}/link
lnameWildcardFilterSearch = search:[::EPSN*,EPSC*]/u/build/repo?path=${application}/copybook/*.cpy

# dependency exclude list filter search
lnameExcludes = ^DFH*,^CEE*,^XCH*,^XWS*,^SQLCA,^DSN*
excludeDependencyFilterSearch = search:[SYSLIB::${lnameExcludes}]/u/build/repo?path=${application}/copybook/

7. Identifying programs impacted by changed copybooks, include files, and statically linked programs

An incremental build uses impact analysis to only build the programs that are out of date or have dependency references (such as copybooks, include files, or macros) that are out of date.

DBB provides the SearchPathImpactFinder class to run an impact analysis on changed copybooks and include files. Whereas the SearchPathDependencyResolver starts with a program and finds all of the copybooks and include files that the program needs to build, the SearchPathImpactFinder class goes the other way. It starts with a changed copybook or include file and finds all of the programs, copybooks and include files that reference it. To do so, it queries the collections in the build metadata database to see what logical files have a dependency reference to the changed copybook or include file being searched. More information on creating the searchPath string is described above.

All of the searches are run against DBB metadata database collections so an active metadata store connection is required. For more information about initializing the MetadataStore utility class, see Metadata store. Also note that the more up-to-date the logical files in the collections to be searched, the more accurate the impact analysis is.

// Start by creating a search path which accepts changed files from '/u/build/repo' and all its sub-directories for files with a '.cpy' file extension 
String searchPath = "search:/u/build/repo?path=**/*.cpy"

// Create a list of collections to search
List<String> collections = new ArrayList<String>()
collection.add("MortgageApplication");

// Instantiate a SearchPathImpactFinder to find programs impacted by changed copybooks 
def finder = new SearchPathImpactFinder(searchPath, collections)

// Find all files impacted by changed copybook 
Set<ImpactFile> findImpacts = finder.findImpactedFiles("copy/epsmtcom.cpy", "/u/build/repo")

The result of running the finder.findImpactedFiles method is a set of impacted files, that is, Set<ImpactFile>. An ImpactFile represents a program or other source file that has a build dependency on the changed file. An ImpactFile contains the following information:

• lname - The logical name of the program. For example, the lname of epsmtinp.cbl is EPSMTINP.
• file - The program's or other source file's path relative to the sourceDir used to scan the file. It is the same value as the file attribute in a logical file.
• language - The language of the source file.
• collection - The name of the collection where the impacted program or source file's logical file is.

Note: The SearchPathImpactFinder.findImpactedFiles method is a recursive search. The list of the impacted files returned by the method contains all the source files that are directly and indirectly impacted by the changed file. For example, when an impact resolution is done on CPYBOOKB, if CPYBOOKB is referenced by CPYBOOKA and if CPYBOOKA is referenced by PRG1, then both CPYBOOKA and PRG1 are in the list of the impacted files for CPYBOOKB.

String impactSearchPath = "search:/u/build/repo?path=**/*.cpy;**/*.bms";

String collectionName = "MortgageApplication.master";
String outputsCollectionName = "MortgageApplication.master.outputs";
List<String> collections = new ArrayList<String>();
collections.add(collectionName);
collections.add(outputsCollectionName);

// create the collections list
String collectionName = "MortgageApplication.master";
String outputsCollectionName = "MortgageApplication.master.outputs";
List<String> collections = new ArrayList<String>();
collections.add(collectionName);
collections.add(outputsCollectionName);

// create and concatenate the search paths
String copybookSearch = "search:[SYSLIB]/u/build/repo?path=MortgageApplication/copybook;MortgageApplication/bms";
String programSearch = "search:[:LINK]/u/build/repo?path=MortgageApplication/cobol";
String impactSearch "$copybookSearch$programSearch;

// create SearchPathImpactFinder and find inpacts
SearchPathImpactFinder finder = new SearchPathImpactFinder(impactSearch, collections)
List<ImpactFile> resolveImpacts = finder.findImpactedFiles("MortgageApplication/cobol/epscmort.cbl", "/u/build/repo");