Build properties
The helloworld.groovy
build script that we write in the tutorial is a simple example of how to use Groovy to build z/OS programs by using DBB APIs. However, it has two problems to be fixed before it can be used for building an enterprise
application:
-
Problem 1: Everything is hardcoded in the script. The program name, the source, the working directories, and the referenced data sets are all defined in the script. Hardcoding not only prevents the build script from building any program other than
helloworld.cbl
, but also potentially ties the build script to a specific build machine where those data sets and directories exist. And only developers and builders who have access to those data sets and directories would be able to run it. -
Problem 2: The script can only build COBOL programs. Most z/OS applications are composed of different types programming languages, such as COBOL, PL/I, Assembler, C, and BMS Maps. Although it is possible to create a single build script that builds any kind of program that it might encounter, the script would be large and contain rather complex conditional logic.
Solutions to Problem 1: Use properties and properties files
Properties are name-value pairs of type string that are defined outside of the build scripts and are either loaded (read) from a file or passed in as command line arguments. Below is a sample properties file:
# Absolute path of the source directory i.e. directory containing .git
sourceDir=/u/app1/src
# Absolute path to the build output directory
workDir=/u/app1/build/work
# High Level Qualifier for build data sets
hlq=APP1.BUILD
# DBB Dependency Data Collection Name
collection=MortgageApplication
Though most scripting languages support passing in string arguments from the command line, depending on the number of properties needed by the build scripts, it might be easier to define the properties in a properties file. However, it is also a common practice to define a properties file as the default properties but also write the build script to allow users to override the default properties with passed-in arguments, as discussed in Passing script arguments from command line.
Categorization of properties
The properties are categorized as follows:
Sandbox properties
In the hello world example, the helloworld.cbl
file was located in the /u/usr1/build
directory. You can tell from the path that the directory probably belongs to a specific user 'usr1'.
- If another user wanted to use the
helloworld.groovy
build script, they must modify it to use their directory instead. - In addition to
/u/usr1/build
being the source directory for the build script, it is also where the build script is copying the SYSPRINT log file to. In general, the work directory is where non-binary outputs of the build process, such as logs, temporary files, and build metadata, are created. - The two data sets that are being written to by the build script have the same high-level qualifier (HLQ) USR1.BUILD.
These three locations comprise what is commonly called the "user sandbox". It is where the user can store modified source code and generate build outputs without overwriting anyone else's files or data set members. The sandbox
properties are good candidates to be included in a properties file. The location of a user's properties file could be passed into the build script as a single argument. A common convention is to have a file called build.properties
in the same directory as the build script that could contain a default sandbox to be used to team builds. The content of which could be:
sourceDir=/u/app1/src
workDir=/u/app1/build/work
hlq=APP1.BUILD
Build properties class
Properties and properties files are fairly common and most scripting languages have native support for them. Groovy, as a Java scripting language, uses the java.util.Properties
class to read properties files. This class is perfectly
adequate to support what has been discussed so far; however, DBB does provide the com.ibm.dbb.build.BuildProperties
class as well. The BuildProperties class has much of the same functionality as the Java Properties class but
it also provides three additional useful features:
-
There is a single static BuildProperties instance. Therefore, the BuildProperties is always available to all scripts without being passed from one script to another.
-
BuildProperties values can reference other build properties. Property references are evaluated dynamically at run time.
-
It introduces a new concept called file properties. File properties are properties that are associated to a set of files or even a single file.
File properties
File properties are useful in cases where some source files being built need to be handled differently than the rest of the source files. An example of this is where some COBOL programs need different compiler options than the rest of
the COBOL programs. Instead of hardcoding the names of the special case files in the build scripts, file properties can be created for those source files in the build.properties
file. This allows for easy maintenance if
you need to add or remove files of this type.
compileOpts=LIB
compileOpts=APOST,DATA31,LIB,DYNAM::App1/cobol/*.cbl,App2/cobol/*.cbl
In the example above, the first property defined is a normal property called compileOpts that is used as the 'default' compile option for the COBOL compiler. The second property defined is a file property. This can be seen by the use of
::
as a delimiter between the property value and the comma-delimited list of file path patterns use to indicate which files the property is associated to. In this case, all of the files in App1/cobol
and App2/cobol
directories with the .cbl file extension are associated with compileOpts=APOST,DATA31,LIB,DYNAM
. All other programs will default to compileOpts=LIB
.
Note: Normally when two properties are defined with the same property name, the second declaration of the property overwrites the first declaration of the property. This rule also applies to the BuildProperties class but only applies for normal key-value pair properties. File properties allow for duplicate property names because the 'key' of a file property is name + file pattern. If two file properties have both the name and a file path pattern that is identical, then the value of the second declaration of the name + file pattern overwrites the first one.
To use the file property in the build script you must first load the build.properties file using a BuildProperties.load(...) method. Then you can use the BuildProperties class name as a static variable in our script:
def opts = BuildProperties.getFileProperty("compileOpts", file)
def compile = new MVSExec().pgm("IGYCRCTL").parm(opts)
In the case where file = App1/cobol/ABXDTC.cbl, then APOST,DATA31,LIB,DYNAM
will be used as the parm value. If the file = App3/cobol/DEBWRD.cbl, then LIB
will be used as the parm value.
DBB configuration properties
Some DBB APIs require you to set configuration like properties on each declaration that are often the same for all declarations and never change during the build process. Examples of this can be seen in the ISPFExec, TSOExec and so on. You can declare DBB configuration properties in the properties files that will automatically set those values without having to code them explicitly in the build scripts.
Below is a list of supported DBB configuration properties:
Property name | Description | Property type |
---|---|---|
dbb.file.tagging | Flag indicating the data set members that are copied to zFS via the CopyToHFS command should be tagged with the CCSID of the file | basic |
dbb.gateway.type | ISPFExec and TSOExec gateway type Possible values are legacy or interactive |
basic |
dbb.gateway.reuseIspfSession | Toggle for the gateway to reuse the ISPF session Applicable for both legacy and interactive gateway |
basic |
dbb.gateway.procedureName | Interactive gateway Procedure Name | basic |
dbb.gateway.accountNumber | Interactive gateway Account Number | basic |
dbb.gateway.groupId | Interactive gateway Group ID | basic |
dbb.gateway.regionSize | Interactive gateway Region Size | basic |
dbb.gateway.logLevel | Interactive gateway Log Level | basic |
dbb.command.reportOnly | Create a build report without running the build. Boolean flag | basic |
dbb.DependencyScanner.allowIncludeInAnyColumn | Toggle for Dependency Scanner | file |
dbb.DependencyScanner.expandIncludeInComment | Toggle for Dependency Scanner | file |
dbb.DependencyScanner.freeFormatCobol | Toggle for Dependency Scanner | file |
dbb.DependencyScanner.languageHint | Toggle for Dependency Scanner | file |
dbb.DependencyScanner.scanNetviewDependencies | Toggle for Dependency Scanner | file |
dbb.DependencyScanner.defaultLibrary | Default Library for Dependency Scanner | file |
dbb.LinkEditScanner.excludeNameFilter | Exclude filter for Link Edit Scanner | file |
dbb.scriptMapping | Script Mapping | file |
Passing script arguments from command line
Another way to customize build scripts is to pass arguments to the script from the command line when invoking the script or in the case of a CI server like Jenkins, there is usually a field in build script plug-in where input arguments are listed. This does not have to be an either/or scenario. A common practice is to have the build script use a properties file as its default properties but also write the build script to allow users to override the default properties with passed in arguments. This is another useful way in handling sandbox properties by having the team build sandbox properties defined in the build.properties file and then allow users to pass in their own sandbox properties for when they want to run a user build.
CliBuilder
Groovy has a utility class called CliBuilder that is very useful for automatically parsing command line arguments which contain both optional and required arguments. It can even output a usage message if desired. For example:
def parseInput(String[] cliArgs){
def cli = new CliBuilder(usage: "deploy.groovy [options]")
cli.b(longOpt:'buztool', args:1, argName:'file', 'Absolute path to UrbanCode Deploy buztool.sh script')
cli.w(longOpt:'workDir', args:1, argName:'dir', 'Absolute path to the DBB build output directory')
cli.c(longOpt:'component', args:1, argName:'name', 'Name of the UCD component to create version in')
cli.h(longOpt:'help', 'Prints this message')
def opts = cli.parse(cliArgs)
if (opts.h) { // if help option used, print usage and exit
cli.usage()
System.exit(0)
}
}
Solutions to Problem 2: Use a main build script to call more dedicated build scripts
Most enterprise applications are comprised of many source files often written in a variety of programming languages, such as COBOL, PL/I, C and Assembler. They may also include specialized source files such as CICS BMS and IMS MFS map files as well as database DBRM files and Link-Edit files. Putting all of the logic to build an application this complex into a single build script would result in a very large and complicated script. A better design would be to have several build scripts with each designed to build a single type of source file. The build process would begin by calling a "main" build script that could perform a number of build initialization tasks such as the following:
- Handling command line arguments
- Loading properties files
- Creating directories and data sets
- Initializing build artifacts
- Scanning files for new dependency data
After the build initialization is complete, the main build script iterates though a build file list, calling each dedicated build script depending on the file type to build. After all the files in the build list have been processed, the main build script could run through a clean up phase, deleting temporary files and finalizing and storing build artifacts.
Build file list
The Mortgage Application sample, which is part of the samples that ship with DBB, contains a build script build.groovy
in its build folder. This is the main build script for the Mortgage Application. It is expecting a file to
be passed in as an argument. This is the name of the file to build. However, if the file has a .txt
file extension, then the build script assumes that it is a text file which contains a list of files that need to be built.
Note: In this documentation and the DBB API descriptions, the term 'file' can have two different meanings depending on the context.
- In some cases, it refers to a physical file on the build machine that has an absolute path, such as
/u/usr1/src/MortgageApplication/build/build.properties
. As when referring to log, properties, text files, or even a source file, that is the source of a CopyToPDS statement. In the DBB API documentation, this usage of 'file' is usually typed as ajava.io.File
. - In other instances as in this case, 'file' refers to a source file that is to be built or scanned. It is the unique ID of the file in the source repository. If the repository is Git, it is the relative path of the file. When loaded from
a Git server onto the local build machine, it is the relative path from the source directory
MortgageApplication/cobol/epsnbrvl.cbl
. When used in this context in the DBB API documentation, the 'file' is usually typed as aString
.
The build list file can be a static file that contains a list of files that are always to be built. Alternatively, it can be dynamically created by a pre-build process to contain only the files that need to be built at that time. An example
of a static build list file is included in the MortgageApplication sample in MortgageApplication/build/files.txt
:
MortgageApplication/bms/epsmlis.bms
MortgageApplication/bms/epsmort.bms
MortgageApplication/cobol/epsmlist.cbl
MortgageApplication/cobol/epsmpmt.cbl
MortgageApplication/cobol/epsnbrvl.cbl
MortgageApplication/cobol_cics/epscsmrd.cbl
MortgageApplication/cobol_cics/epscsmrt.cbl
MortgageApplication/cobol_cics_db2/epscmort.cbl
MortgageApplication/link/epsmlist.lnk
MortgageApplication/mfs/dfsiv1.mfs
MortgageApplication/copybook/epsmortf.cpy
MortgageApplication/copybook/epsmtcom.cpy
MortgageApplication/copybook/epsmtinp.cpy
MortgageApplication/copybook/epsmtout.cpy
MortgageApplication/copybook/epsnbrpm.cpy
MortgageApplication/copybook/epspdata.cpy
Since this file is used for both dependency scanning and build programs, the list contains both programs and copybooks. However, only the programs and map files will be built. This is done by using script mappings.
Script mappings
Script mappings are file properties in which the property name is scriptMapping
and the value is the name of the specific build script that should be called to build the files associated to the property. Script mapping supports
specification of criteria such as file extensions, file paths, glob and regex patterns, for example:
scriptMapping = BMSProcessing :: MortgageApplication/bms/*.bms
scriptMapping = Compile :: MortgageApplication/cobol/epsmlist.cbl, MortgageApplication/cobol/epsnbrvl.cbl
scriptMapping = CobolCompile :: MortgageApplication/cobol/epsmpmt.cbl, MortgageApplication/cobol_cics/*.cbl, MortgageApplication/cobol_cics_db2/*.cbl
scriptMapping = LinkEdit :: MortgageApplication/link/*.lnk
scriptMapping = MFSGENUtility :: MortgageApplication/mfs/*.mfs
These properties are loaded at the beginning of the build process with the other build properties. You can see the referred called build scripts in the MortgageApplication/build
folder:
BMSProcessing.groovy
CobolCompile.groovy
Compile.groovy
LinkEdit.groovy
MFSGENUtility.groovy
Though script mappings can be accessed as file properties using the DBB BuildProperties class, DBB also provides a special utility class called com.ibm.dbb.build.ScriptMappings
that contains additional methods for processing script
mappings. It contains useful methods such as isMapped(file)
, getScriptName(file)
and getMappedList(scriptName, buildList)
which returns a sublist of all the files in the build list that are mapped to
a specific script. An example of using ScriptMappings.getMappedList(scriptName, buildList)
to get a list of files mapped to specific script and the iterate over the list calling the build script for each file can be seen in
MortgageApplication/build/build.groovy
:
// Use the ScriptMappings class to get the files mapped to the build script
def scriptList = ScriptMappings.getMappedList(script, buildList)
def scriptName = "$properties.sourceDir/MortgageApplication/build/${script}.groovy"
scriptList.each { file ->
run(new File(scriptName), [file] as String[])
processCounter++
}
Reducing time consumption by Groovy build script caching
During build execution, the called Groovy scripts are compiled immediately into Java classes and invoked. Though the compilation is fast, it still takes time and if the build process builds hundreds or even thousands of source files, this
can result in significant build processing times. Since most build processes generally call the same few Groovy scripts over and over again, adding a Groovy script caching mechanism to your build process can greatly reduce the amount of
time it takes to load, recompile and execute Groovy scripts resulting in lower process times. DBB provides a Groovy base script class com.ibm.dbb.groovy.ScriptLoader
that you can extend when writing your build scripts that will
automatically load and cache called Groovy scripts.
Script caching API
You can extend your build scripts by adding the following line to the beginning of the script:
@groovy.transform.BaseScript com.ibm.dbb.groovy.ScriptLoader baseScript
This now provides the following built-in methods that the build script can now invoke:
void runScript(File script, String[] args=[])
- This method is used to call another executable Groovy script.- Will automatically try to use the cached version of the compiled script class or updates the cache if the script is being called for the first time during the build process.
- Can be used to call both DBB ScriptLoader based Groovy scripts and generic Groovy scripts
void runScript(File script, Map<String,Object> argMap)
- This method is similar to the previous method but passes an argument Map instead of a String array.- For use only when calling another DBB ScriptLoader based Groovy script
- Allows for the easy passing of objects between scripts
- A Map property called
argMap
is created in the target script
- If the target script is not a DBB ScriptLoader based Groovy script, then an empty String [] args is passed instead.
- For use only when calling another DBB ScriptLoader based Groovy script
GroovyObject loadScript(File script)
- This method performs automatic caching similar to the other methods described but only loads the script without trying to run it.- This method can be used to load object-oriented Groovy scripts, that is, Tools.groovy
getScriptDir()
- Returns the String representation of the parent directory of the running script.
Note: The File script
argument in the above methods supports both absolute and relative paths. If the script parameter contains a relative path, then it is automatically appended to the current script's parent
directory. This means that loadScript(new File("Tools.groovy"))
will try to load the Tools.groovy script file in the same directory that the current running script is in.
Example:
@groovy.transform.BaseScript com.ibm.dbb.groovy.ScriptLoader baseScript
import com.ibm.dbb.build.*
// receive passed arguments from argument map
def file = argMap.file
. . .
// call the bind package script using script caching API and passing argument hashmap
runScript(new File("BindPackage.groovy"), ["file":file])