DataStage command-line tools
CPDCTL and the dsjob
tool are command-line interfaces (CLI) you can use
to manage your DataStage® resources in IBM Cloud Pak for
Data.
- CPDCTL:
cpdctl dsjob
orcpdctl datastage
- Projects
- Jobs
- Job logs
- Job migration
- Flows
- Hardware specifications
- Runtime environments
- Connections
- Imports and exports
- .zip files
- Subflows
- Parameter sets
- Table definitions
- XML libraries
- Environment variables
- Job runs
- Versions
- User-defined stages
- Assets and attachments
- Match specifications
- Standardization rules
- Pipelines
Resources
For installation, configuration, available commands, supported outputs, and usage scenarios, refer to IBM Cloud Pak for Data Command Line Interface.
For more information on the CPDCTL command, see IBM Cloud Pak for Data Command Line Interface.
For detailed information about installing, configuring, and using the DataStage jobs command-line interface, see Installation.
Commands
cpdctl dsjob
commands, you must set the environment variable
CPDCTL_ENABLE_DSJOB to true
in the environment where the IBM Cloud Pak® for Data command-line interface is installed.dsjob
command line environment, you must
escape any special characters ($, ") in your password with a forward slash. For
example, myPa$$word
must be written as
myPa\$\$word
.- Listing projects
- The following syntax displays a list of all known projects on the specified project:
cpdctl dsjob list-projects [--sort|--sort-by-time] [--with-id] [flags]
with-id
when specified prints the project id and project name.sort
when specified returns the list of projects sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.
A list of all the projects is displayed, one per line.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating projects
- The following syntax is used to create a project:
cpdctl dsjob create-project -n NAME [flags]
name
is the name of the project that is being created.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting projects
- The following syntax is used to delete a project:
cpdctl dsjob delete-project {--project PROJECT | --project-id PROJID}
project
is the name of the project that is being deleted.project-id
is the id of the project that is being deleted. One ofproject
orproject-id
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing jobs
- The following syntax displays a list of all jobs in the specified project:
cpdctl dsjob list-jobs {--project PROJECT | --project-id PROJID} [--sort] [--with-id]
project
is the name of the project that contains the jobs to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified.sort
when specified returns the list of jobs sorted in alphabetical order. This field is optional.with-id
when specified prints the job id along with the name of the job.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating jobs
- The following syntax creates a job in the specified project:
cpdctl dsjob create-job {--project PROJECT | --project-id PROJID} {--flow NAME | --flow-id ID} [--name NAME] [--description DESCRIPTION] [--schedule-start yyyy-mm-dd:hh:mm] [--schedule-end yyyy-mm-dd:hh:mm] [--repeat every/hourly/daily/monthly --minutes (0-59) --hours (0-23) --day-of-week (0-6) --day-of-month (1-31)]
project
is the name of the project that the job is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job to be created.description
is the description of the job to be created. This field is optional.flow
is the name of the flow. This field must be specified.repeat
indicates frequency of job run. Permitted values areevery
,hourly
,daily
,weekly
, andmonthly
. The default value isnone
.minutes
indicates interval in minutes or the minutes at which to run the job. Values in the range 0-59 are accepted.hours
indicates hour of the day at which to run the job. Values in the range 0-23 are accepted.day-of-month
repeats on day of the month, works with minutes and hours. Values in the range 0-31 are accepted. Ex: 2 (runs on the second of the month).schedule-start
is the starting time for scheduling a job.schedule-end
is the ending time for scheduling a job.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting jobs
- The following syntax fetches a job by name from the specified project:
cpdctl dsjob get-job {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that contains the job.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the queried job.id
is the id of the job. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting jobs
- The following syntax deletes a job by name from the specified project:
cpdctl dsjob delete-job {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the job.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job that is being deleted.id
is the id of the job. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Displaying job information
- The following syntax displays the available information about a specified job:
cpdctl dsjob jobinfo {--project PROJECT | --project-id PROJID} {--job NAME | --job-id ID} [--full]
project
is the name of the project that contains the job.project-id
is the id of the project. One ofproject
orproject-id
must be specified.job
is the name of the job.job-id
is the id of the job. One ofjob
orjob-id
must be specified.full
displays more detailed information about the job, including information about all job runs. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Running jobs
- You can use the run command to start, stop, validate, and reset jobs. The run operation is
asynchronous in nature and the status code indicates whether the job run
is successfully submitted or not except when the --wait option is
specified. Please see --wait flag description on how the behavior changes.
cpdctl dsjob run {--project PROJECT | --project-id PROJID} {--job NAME | --job-id ID} [--param PARAM] [--param-file FILENAME] [--env ENVJSON] [--wait secs]
project
is the name of the project that contains the job.project-id
is the id of the project. One ofproject
orproject-id
must be specified.job
is the name of the job.job-id
is the id of the job. One ofjob
orjob-id
must be specified.param
specifies a parameter value to pass to the job. The value is in the formatname=value
, where name is the parameter name and value is the value to be set. This flag can be repeated, ex:--param k1=v1 --param k2=v2
paramfile
specifies a file that contains the parameter values to pass to the job. This field is not implemented currently.env
specifies the environment in which job is run.env
is specified as a key=value pair. Key `env` or `env-id` can be used to chose a runtime environment. Example:--env $APT_DUMP_SCORE=true --env env=ds-px-default
wait
the job run waits for the specified amount of time for the job to finish. The job logs are printed to the output until the job is completed or the wait time expires. The return status indicates whether the job has finished, finished with warning, raised an error, or timed out after waiting. This field is optional.
When the
job
parameter starts with a `$` it will also be added as a environment variable.A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Stopping jobs
- You can use the stop command to stop or cancel running jobs. The stop operation is
asynchronous in nature and the status code indicates whether the job
stop is successfully submitted or not.
cpdctl dsjob stop {--project PROJECT | --project-id PROJID} {--job NAME | --job-id ID} [--run-id RUNID]
project
is the name of the project that contains the job.project-id
is the id of the project. One ofproject
orproject-id
must be specified.job
is the name of the job.job-id
is the id of the job. One ofjob
orjob-id
must be specified.runid
can be specified to cancel or stop an existing job run. Ifrunid
is not specified, therunid
of the latest job run that is not completed is used by default. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Displaying a specific log entry
- The following syntax displays the specified entry in a job log file:
cpdctl dsjob logdetail {--project PROJECT | --project-id PROJID} {--job NAME | --job-id ID} [--run-id RUNID] [--eventrange EVENTRANGE] [--compatible]
project
is the name of the project that contains the job with the specified log entry.project-id
is the id of the project. One ofproject
orproject-id
must be specified.job
is the name of the job.job-id
is the id of the job. One ofjob
orjob-id
must be specified.runid
processes the log entry for a specificrunid
. Ifrunid
is not specified, the latest run is used by default. This field is optional.eventrange
is the range of event numbers that is assigned to the entry that is printed to the output. The first entry in the file is 0. Ifeventrange
is not specified, the full log is processed. For example, if you specifyeventrange 2-4
, the third, fourth, and fifth entries from the log are printed.compatible
will output logs in the format previously used by DataStage components. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Displaying a short log entry
- The following syntax displays a summary of entries in a job log file:
cpdctl dsjob logsum {--project PROJECT | --project-id PROJID} {--job NAME | --job-id ID} [--run-id RUNID] [--type TYPE] [--max MAX] [--compatible]
project
is the name of the project that contains the job with the log entries that are being retrieved.project-id
is the id of the project. One ofproject
orproject-id
must be specified.job
is the name of the job.job-id
is the id of the job. One ofjob
orjob-id
must be specified.runid
processes the log entry for a specific runid. Ifrunid
is not specified, the latest run is used by default. This field is optional.type
specifies the type of log entry to retrieve. Iftype
is not specified, all the entries are retrieved.type
can be one of the following options:- INFO: Information
- WARNING: Warning
- FATAL: Fatal error
- REJECT: Rejected rows from a Transformer stage
- STARTED: All control logs
- RESET: Job reset
- BATCH: Batch control
- ANY: All entries of any type. This option is the default if
type
is not specified.
compatible
will output logs in the format previously used by DataStage components. This field is optional.max n
limits the number of entries that are retrieved ton
.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Identifying the newest log entry
- The following syntax displays the ID of the newest log entry of the specified type:
cpdctl dsjob lognewest {--project PROJECT | --project-id PROJID} {--job NAME | --job-id ID} [--run-id RUNID] [--type TYPE]
project
is the name of the project that contains the job with the log entry that is being retrieved.project-id
is the id of the project. One ofproject
orproject-id
must be specified.job
is the name of the job.job-id
is the id of the job. One ofjob
orjob-id
must be specified.type
can be one of the following options:- INFO: Information
- WARNING: Warning
- FATAL: Fatal error
- REJECT: Rejected rows from a Transformer stage
- STARTED: All control logs
- RESET: Job reset
- BATCH: Batch control
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Managing job migrations
- The Migrate command can be used to create data flows from an exported legacy ISX file. You can
use the command to check status or cancel a migration that is in progress.
cpdctl dsjob migrate {--project PROJECT | --project-id PROJID} [--on-failure ONFAILURE] [--conflict-resolution CONFLICT-RESOLUTION] [--attachment-type ATTACHMENT-TYPE] [--import-only] [--create-missing] [--file-name FILENAME] [--status IMPORT-ID --format csv/json] [--stop IMPORT-ID]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.on-failure
indicates what action to taken if the import process fails. Possible options are eithercontinue
orstop
. This field is optional.conflict-resolution
or-action
specify the resolution when the data flow to be imported has a name conflict with an existing data flow in the project or catalog. Possible resolutions areskip
,rename
, orreplace
. This field is optional.attachment-type
is the type of attachment. The default attachment type isisx
. This field is optionalimport-only
when specified imports flows without compiling them or creating a job.create-missing
when specified creates missing parameter sets and job parameters.body
is the file name of the import file. This field is required for an import operation but not used with options -stop or -status.file-name
is the name of the input file. This field is required for an import operation but not with options-stop
or-status
.status
returns the status of a previously submitted import job. A value forimportid
must be specified with this option.stop
cancels an import operation that is in progress. A value forimportid
must be specified with this option.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing flows
- The following syntax displays a list of all flows in the specified project:
cpdctl dsjob list-flows {--project PROJECT | --project-id PROJID} [--sort|--sort-by-time] [--with-id]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.sort
when specified returns the list of flows sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the flow id along with the name of the flow.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating flows
- The following syntax creates a flow in the specified project:
cpdctl dsjob create-flow {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--pipeline-file FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the flow that is being created.description
is the description of the flow that is being created. This field is optional.pipeline-file
is the name of the file that contains the flow JSON. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting flows
- The following syntax fetches a flow by name from the specified project:
cpdctl dsjob get-flow {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the queried flow.id
is the id of the flow. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting flows
- The following syntax deletes a flow by name from the specified project:
cpdctl dsjob delete-flow {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the flow.id
is the id of the flow. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Compiling flows
- The following syntax allows you to compile flows in the specified project:
cpdctl dsjob compile {--project PROJECT | --project-id PROJID} [{--name NAME | --id ID}] [--osh] [--threads <n>]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the flow.id
is the id of the flow. One ofname
orid
can be specified. If not present, all the flows in the project are compiled.osh
the output will display compiled 'osh' output. This field is optional.threads
specifies the number of parallel compilations to run. The value should be in the range 5-20, default value is 5. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing hardware specifications
- The following syntax displays a list of all hardware specifications in the specified project:
cpdctl dsjob list-hardware-spec {--project PROJECT | --project-id PROJID} [--type TYPE] [--full FULL] [--sort|--sort-by-time] [--with-id]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the DataStage Hardware Specifications in the project are displayed, one per line.type
displays list of all hardware specifications in the project for the specified type, ex: DataStage, Spark, and Nodes.full
provides full configuration details of each hardware specification. This field is optional.sort
when specified returns the list of hardware specifications sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the hardware specification id along with the name of the hardware specification.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating hardware specifications
- The following syntax creates a hardware specification for the specified project:
cpdctl dsjob create-hardware-spec {--project PROJECT | --project-id PROJID} [[--name NAME] [--description DESCRIPTION] [--body BODY-JSON]] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the hardware specification being created.description
is the description of the hardware specification being created. This field is optional.body
should contain the hardware specification in json format. Alternatively, the hardware specification can be provided in a file by using--filename
.filename
is the name of the file that contains the hardware specification. Alternatively, the hardware specification can be provided inline by using --body. Either --body or --filename must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting hardware specifications
- The following syntax fetches a hardware specification by name from the specified project:
cpdctl dsjob get-hardware-spec {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the hardware specification.id
is the id of the hardware specification. One ofname
orid
must be specified.file-name
is the name of the output file to which the hardware specification is written.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing runtime environments
- The following syntax displays a list of all Environments in the specified project:
cpdctl dsjob list-envs {--project PROJECT | --project-id PROJID} [--types TYPE] [--full] [--sort] [--with-id]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the DataStage Environments in the project are displayed, one per line.type
displays list of all environments in the project specified by the type. The value should be one ofnotebook
,wml_flow
,rstudio
,default_spark
,remote_spark
,jupyterlab
,remote_yarn
,datastage
,profiling
,modeler
, ordata_privacy
. This field is optional.full
provides full configuration details of each environment. This field is optional.sort
when specified returns the list of environments sorted in alphabetical order. This field is optional.with-id
when specified prints the environment id along with the name of the environment.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating runtime environments
- The following syntax creates a runtime environment for the specified project:
pdctl dsjob create-env {--project PROJECT | --project-id PROJID} [--name NAME] [--display-name DISPLAY-NAME] [--type TYPE] [--location LOCATION] [--hwspec HWSPEC-NAME]] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the environment that is being created. Used whenfile-name
is not specified.display-name
is the long name of the environment being created. Used whenfile-name
is not specified.type
is the type of environment to create, ex:datastage
. Used whenfilename
is not specified.location
if specified is the json-formatted location information needed to access the environment. Used whenfilename
is not specified.hwspec
is the name of the hardware specification used to create the environment. Used whenfilename
is not specified.file-name
is the name of the file that contains the hardware specification, location and other attributes. When specified, all other options are ignored.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting runtime environments
- The following syntax fetches a runtime environment by name from the specified project:
cpdctl dsjob get-env {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--file-name FILENAME] [--with-metadata]
project
is the name of the project that the environment is fetched from.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the queried environment.id
is the id of the environment. One ofname
orid
must be specified.file-name
is the name of the output file to which the environment is written.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing connections
- The following syntax displays a list of all connections in the specified project:
cpdctl dsjob list-connections {--project PROJECT | --project-id PROJID} [--sort|--sort-by-time] [--with-id]
project
is the name of the project that contains the connections to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the DataStage connections in the project is displayed, one per line.sort
when specified returns the list of connections sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the connection id along with the name of the connection.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating connections
- The following syntax creates a flow in the specified project:
cpdctl dsjob create-connection {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--datasource-type TYPE] [--country COUNTRY] [--property-file FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the connection being created.description
is the description of the connection being created. This field is optional.datasource-type
is the data source type for the connection ex: 971223d3-093e-4957-8af9-a83181ee9dd9.country
is the country of origin for the connection. The default is "us."property-file
is the name of the file that contains the connection properties. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting connections
- The following syntax fetches a connection by name from the specified project:
cpdctl dsjob get-connection {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that the connection is fetched from.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the queried connection.id
is the id of the connection. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
is the name of the output file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting connections
- The following syntax deletes a flow by name from the specified project:
cpdctl dsjob delete-connection {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that the connection is deleted from.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the connection.id
is the id of the connection. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing imports
- The following syntax displays a list of all imports into the specified project:
cpdctl dsjob list-imports {--project PROJECT | --project-id PROJID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Importing
- The following syntax imports the specified project to a file:
cpdctl dsjob import {--project PROJECT | --project-id PROJID} --import-file FILENAME
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.import-file
is the name of the file that contains previously exported assets.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Exporting
- The following syntax exports the specified project to a file:
cpdctl dsjob export {--project PROJECT | --project-id PROJID} [--name NAME] [--description DESCRIPTION] [--export-file FILENAME] [--wait secs] [--asset-type TYPE] [--asset <name,type>...] [--all]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.asset list
is a list of all the asset names to be exported. Format:--asset type=assetname1,assetname2
.name
is the name of the export.asset-type
is a list of all asset types to export, ex:--asset-type Connection --asset-type data_flow
.description
is a description of the exported assets.export-file
is the file for assets to be exported to.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing exports
- The following syntax displays a list of all exports from the specified project:
cpdctl dsjob list-exports {--project PROJECT | --project-id PROJID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Saving exports
- The following syntax saves an export to a file.
cpdctl dsjob save-export {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} --export-file FILENAME
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the export.id
is the id of the export. One ofname
orid
must be specified.export-file
is the name of the file that the export is saved to.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting exports
- The following syntax deletes an export from the specified project:
cpdctl dsjob delete-export {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the export.id
is the id of the export. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Importing .zip files
- The following syntax imports DataStage flows from a .zip file into a specified project:
cpdctl dsjob import-zip {--project PROJECT | --project-id PROJID} [--on-failure ONFAILURE] [--conflict-resolution CONFLICT-RESOLUTION] [--file-name FILE-NAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.on-failure
indicates what action to take if the import fails. This field is optional. The default option is continue, the other option is stop.conflict-resolution
specifies the resolution when the data flow to be imported has the same name as an existing data flow in the project or catalog. This field is optional. The default option is skip, the others are rename and replace.file-name
is the name of the .zip file that contains all the DataStage flows and DataStage components to be imported. This field is mandatory.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting status of import requests from .zip files
- The following syntax gets the status of an import request using import-zip:
cpdctl dsjob get-import-zip {--project PROJECT | --project-id PROJID} --import-id [--format json|csv] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.import-id
is the id of the import. This field is mandatory.format
specifies the format of the output file. This field is optional. The default value is json. The other option is csv.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Exporting .zip files
- The following syntax exports a DataStage flow and dependent DataStage components to a .zip file
from a specified project:
cpdctl dsjob export-zip {--project PROJECT | --project-id PROJID} {--name FLOW | --id ID} [--file-name FILENAME] [--no-secrets] [--no-deps]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the flow.id
is the id of the flow. One ofname
orid
must be specified.no-deps
specifies that flow dependencies will not be exported. This field is optional. The default option is false.no-secrets
specifies that secrets will not be exported. This field is optional. The default option is false.file-name
specifies the name of the .zip file to which the flow is exported.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing subflows
- The following syntax displays a list of all subflows in the specified project:
cpdctl dsjob list-subflows {--project PROJECT | --project-id PROJID} [--sort|--sort-by-time] [--with-id]
project
is the name of the project that contains the subflows to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the subflows in the project is displayed, one per line.sort
when specified returns the list of subflows sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the subflow id along with the name of the subflow.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating subflows
- The following syntax creates a subflow in the specified project:
cpdctl dsjob create-subflow {--project PROJECT | --project-id PROJID} --name NAME [--pipeline-file FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the subflow.id
is the id of the subflow. One ofname
orid
must be specified.pipeline-file
is the name of the file that contains the subflow JSON. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting subflows
- The following syntax fetches a subflow by name from the specified project:
cpdctl dsjob get-subflow {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output file|json] [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the subflow.id
is the id of the subflow. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting subflows
- The following syntax deletes a subflow by name from the specified project:
cpdctl dsjob delete-subflow {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the subflow.id
is the id of the subflow. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing ParameterSets
- The following syntax displays a list of all ParameterSets in the specified project:
cpdctl dsjob list-paramsets {--project PROJECT | --project-id PROJID} [--sort|--sort-by-time] [--with-id]
project
is the name of the project that contains the ParameterSets to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the ParameterSets in the project is displayed, one per line.sort
when specified returns the list of ParameterSets sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the parameter set id along with the name of the parameter set.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating ParameterSets
- The following syntax creates a ParameterSet in the specified project:
cpdctl dsjob create-paramset {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--file-name FILENAME] [--param type:name:[value]...]
project
is the name of the project that contains the ParameterSets to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the ParameterSet being created. This field is mandatory.description
is a detailed description of the ParameterSet being created. This field is optional.file-name
is the name of the file that contains the definitions of the ParameterSet.param
is used to specify parameters at the command line instead of using thefile-name
option.
One offile-name
orparam
must be specified. Sample file content shown below:[ { "name": "parm3", "prompt": "parm3", "type": "int64", "value": 33 }, { "name": "parm33", "prompt": "parm33", "type": "string", }, { "name": "parm333", "prompt": "parm333", "type": "Float64", "value": 33.3 } ]
Sample command line content usingparam
shown below:
The format is type:name:value and type may be one of time, timestamp, date, int64, sfloat, string, list, path."--param", "int64:parm1:12", "-param", "sfloat:parm2:12.3", "-param", "string:parm3:abc", "-param", "time:ptime:12:12:12", "-param", "date:pdate:11/06/2021", "-param", "timestamp:pts:11/06/2021:12:12:12"
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting ParameterSets
- The following syntax fetches a ParameterSet by name from the specified project:
cpdctl dsjob get-paramset {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output file|json] [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the ParameterSet.id
is the id of the ParameterSet. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting ParameterSets
- The following syntax deletes a ParameterSet by name from the specified project:
cpdctl dsjob delete-paramset {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the ParameterSet.id
is the id of the ParameterSet. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing TableDefs
- The following syntax displays a list of all TableDefs in the specified project:
cpdctl dsjob list-tabledefs {--project PROJECT | --project-id PROJID} [--sort|--sort-by-time] [--with-id]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the TableDefs in the project is displayed, one per line.sort
when specified returns the list of TableDefs sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the table definition id along with the name of the table definition.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating TableDefs
- The following syntax creates a TableDef in the specified project:
cpdctl dsjob create-tabledef {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--file-name FILENAME] [--column type:name:columnattribute...]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the TableDef being created. This field is mandatory.description
is a detailed description of the TableDef being created. This field is optional.file-name
is the name of the file that contains the definitions of the Table.column
is used to specify columns at the command line instead of using thefile-name
option.
One offile-name
orcolumn
must be specified. Sample file content shown below:{ "column_info": {}, "data_asset": { "additionalProperties": {}, "columns": [ { "name": "CreditCardID", "type": { "length": 10, "nullable": false, "scale": 0, "signed": true, "type": "INTEGER" } }, ... more columns follow ], "dataset": true, "mime_type": "application/json" }, "data_definition": {}, "ds_info": {} }
Sample command line content usingcolumn
shown below:
The format is type:name:value and type may be one of Date, Decimal, General, Numeric, String, Time, Timestamp."--column", "Numeric:parm1:length=6", "--column", "Decimal:parm2:scale=2", "--column", "String:parm3:nullable=true,length=120", "--column", "Time:ptime", "--column", "Date:pdate", "--column", "Timestamp:pts:nullable=true""
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting TableDefs
- The following syntax fetches a TableDef by name from the specified project:
cpdctl dsjob get-tabledef {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output file|json] [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the TableDef.id
is the id of the TableDef. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting TableDefs
- The following syntax deletes a TableDef by name from the specified project:
cpdctl dsjob delete-tabledef {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the TableDef.id
is the id of the TableDef. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing XML libraries
- The following syntax displays a list of all XML libraries in the specified project:
cpdctl dsjob list-libraries {--project PROJECT | --project-id PROJID} [--sort|--sort-by-time] [--with-id]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the XML libraries in the project is displayed, one per line.sort
when specified returns the list of XML libraries sorted in alphabetical order. This field is optional.sort-by-time
when specified sorts the list by create or update time. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the XML library id along with the name of the XML library.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating XML libraries
- The following syntax creates an XML library in the specified project:
cpdctl dsjob create-library {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--schema-type TYPE] [--folder FOLDER]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the XML library that is being created. This field is mandatory.description
is detailed description of the XML Library that is being created. This field is optional.schema-type
is the name of the XML Library scheme. This field is optional. The default value is XML, the other options are JSON, JSD and JSN.folder
is the name of the folder for the XML library that is being created.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting XML libraries
- The following syntax fetches an XML library by name from the specified project:
cpdctl dsjob get-library {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the XML library.id
is the id of the XML library. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting XML libraries
- The following syntax deletes a library by name from the specified project:
cpdctl dsjob delete-library {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the XML library.id
is the id of the XML library. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Uploading an XML library file
- The following syntax uploads a .zip file to an existing DataStage XML schema library by name in
a specified project:
cpdctl dsjob upload-library-file {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} --file-name FILENAME
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the XML library.id
is the id of the XML library. One ofname
orid
must be specified.file-name
is the name of the .zip file that contains the schema definitions. This field is mandatory.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Downloading an XML library file
- The following syntax downloads a .zip file of an existing DataStage XML schema library by name
in a specified project:
cpdctl dsjob download-library-file {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} --file-name FILENAME
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the XML library.id
is the id of the XML library. One ofname
orid
must be specified.file-name
is the name of the output file that the XML library's schema definitions are written in. This field is mandatory.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting an XML library file
- The following syntax deletes a set of files from an existing DataStage XML schema library by
name in a specified project:
cpdctl dsjob delete-library-file {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} --file-name FILENAME
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the XML library.id
is the id of the XML library. One ofname
orid
must be specified.file-name
is the names of the files to be deleted from the XML library. This field is mandatory.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting job run statistics
- The following syntax gets job run statistics for a particular job run in a project:
cpdctl dsjob jobrunstat {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--run-id RUNID] [--all]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job.id
is the id of the job. One ofname
orid
must be specified.run-id
is the run id of the particular job run. This field is optional, if omitted the last job run statistics are displayed.all
causes the statistics for all runs for the job to be displayed. When using this flagrun-id
is ignored.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Pruning job runs
- Prune commands can be used to delete the job runs based on age or number of runs. The following
syntax can be used to prune job runs in a project:
cpdctl dsjob prune {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--keep-runs NUMRUNS] [--keep-days NUMDAYS] [--threads n] [--dry-run]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job.id
is the id of the job. One ofname
orid
must be specified.keep-runs
specifies the number of latest runs to keep and deletes rest of the job runs clearing up space.keep-days
specifies a number of days and deletes all job runs older than that number.threads
specifies the number of parallel concurrent cleanup routines to run with one per job. The value should be in the range 5-20, default value is 5. This field is optional.dry-run
does a mock run without deleting the job runs.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting job run cleanup
- It is possible to have job runs that never complete and remain stale. These jobs re stuck in a
starting or running state. The following syntax cleans up job runs in a project:
cpdctl dsjob jobrunclean {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--run-id RUNID] [--dry-run] [--threads n]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job.id
is the id of the job. One ofname
orid
must be specified.run-id
is the run id of the particular job run to clean up. This field is optional.threads
specifies the number of parallel concurrent cleanup routines to run with one per job. The value should be in the range 5-20, default value is 5. This field is optional.dry-run
does a mock run without deleting the job runs.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing job runs
- The following syntax lists job runs for the specified job:
cpdctl dsjob list-jobruns {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--detail] [--output file|json] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job.id
is the id of the job. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting job runs
- The following syntax gets job run details from the specified job:
cpdctl dsjob get-jobrun {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--run-id RUNID] [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the job.id
is the id of the job. One ofname
orid
must be specified.run-id
is the id of the job run.output
specifies the format of the output. You can generate a JSON or output to a file. This field is optional.file-name
specifies the name of the file to which the output is written. If not specified, the job run id is used as the name.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing environment variables
- The following syntax lists all environment variables defined in a specified project environment:
cpdctl dsjob list-env-vars {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--type TYPE]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the environment.id
is the id of the environment. One ofname
orid
must be specified.type
is the type of the environment.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Updating environment variables
- The following syntax updates the environment variables defined in a specified project environment:
cpdctl dsjob update-env-vars {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--type TYPE] [--env k=v...] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the environment.id
is the id of the environment. One ofname
orid
must be specified.type
is the type of the environment.env
is a list of environment variables and their values to update or create.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting environment variables
- The following syntax deletes the environment variables that are defined in a specified project environment:
cpdctl dsjob delete-env-vars {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--type TYPE] [--env ENV...] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the environment.id
is the id of the environment. One ofname
orid
must be specified.type
is the type of the environment.env
is the name of the environment variable to delete. List all variables to delete: the field is repeated.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Printing versions
- The following command prints all the versions of the DataStage components that are installed in
the cluster.
cpdctl dsjob version
- Listing build stages
- The following syntax displays a list of all build stages in the specified project:
cpdctl dsjob list-build-stages {--project PROJECT | --project-id PROJID} [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the build stages to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the build stages in the project is displayed, one per line.sort
when specified returns the list of build stages sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of build stages will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the build stage id along with the name of the build stage.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating build stages
- The following syntax creates a build stage in the specified project:
cpdctl dsjob create-build-stage {--project PROJECT | --project-id PROJID} [--file-name ENTITYFILE]
project
is the name of the project that the build stage is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.file-name
is the name of the file that contains the build stage JSON. The JSON contains the name, description and all other stage details. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting build stages
- The following syntax fetches a build stage by name from the specified project:
cpdctl dsjob get-build-stage {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that contains the build stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the build stage.id
is the id of the build stage. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting build stages
- The following syntax deletes a build stage by name from the specified project:
cpdctl dsjob delete-build-stage {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the build stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the build stage that is being deleted.id
is the id of the build stage. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Generating build stages
- The following syntax generates and compiles the code for a build stage:
cpdctl dsjob generate-buildop {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the build stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the build stage that is being generated.id
is the id of the build stage. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing custom stages
- The following syntax displays a list of all custom stages in the specified project:
cpdctl dsjob list-custom-stages {--project PROJECT | --project-id PROJID} [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the custom stages to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the custom stages in the project is displayed, one per line.sort
when specified returns the list of custom stages sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of custom stages will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the custom stage id along with the name of the custom stage.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating custom stages
- The following syntax creates a custom stage in the specified project:
cpdctl dsjob create-custom-stage {--project PROJECT | --project-id PROJID} [--file-name ENTITYFILE]
project
is the name of the project that the custom stage is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.file-name
is the name of the file that contains the custom stage JSON. The JSON contains the name, description and all other stage details. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting custom stages
- The following syntax fetches a custom stage by name from the specified project:
cpdctl dsjob get-custom-stage {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that contains the custom stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the custom stage.id
is the id of the custom stage. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting custom stages
- The following syntax deletes a custom stage by name from the specified project:
cpdctl dsjob delete-custom-stage {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the custom stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the custom stage that is being deleted.id
is the id of the custom stage. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing wrapped stages
- The following syntax displays a list of all wrapped stages in the specified project:
cpdctl dsjob list-wrapped-stages {--project PROJECT | --project-id PROJID} [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the wrapped stages to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the wrapped stages in the project is displayed, one per line.sort
when specified returns the list of wrapped stages sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of wrapped stages will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the wrapped stage id along with the name of the wrapped stage.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating wrapped stages
- The following syntax creates a wrapped stage in the specified project:
cpdctl dsjob create-wrapped-stage {--project PROJECT | --project-id PROJID} [--file-name ENTITYFILE]
project
is the name of the project that the wrapped stage is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.file-name
is the name of the file that contains the wrapped stage JSON. The JSON contains the name, description and all other stage details. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting wrapped stages
- The following syntax fetches a wrapped stage by name from the specified project:
cpdctl dsjob get-wrapped-stage {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that contains the wrapped stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the wrapped stage.id
is the id of the wrapped stage. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting wrapped stages
- The following syntax deletes a wrapped stage by name from the specified project:
cpdctl dsjob delete-wrapped-stage {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the wrapped stage.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the wrapped stage that is being deleted.id
is the id of the wrapped stage. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing assets
- The following syntax displays a list of all assets in the specified project:
cpdctl dsjob list-assets {--project PROJECT | --project-id PROJID} [-asset-type ASSETTYPE] [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the assets to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the assets in the project is displayed, one per line.asset-type
is the type of asset to list. If not specified, all assets are listed.sort
when specified returns the list of assets sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of assets will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the asset id along with the name of the asset.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating assets
- The following syntax creates a asset in the specified project:
cpdctl dsjob create-asset {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--asset-type ASSETTYPE] [--file-name ENTITYFILE]
project
is the name of the project that the asset is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the asset. This field must be specified.description
is the description of the asset.asset-type
is the type of the asset. This field must be specified.file-name
is the name of the file that contains the asset JSON. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting assets
- The following syntax fetches a asset by name from the specified project:
cpdctl dsjob get-asset {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--asset-type ASSETTYPE] [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that contains the asset.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the asset.id
is the id of the asset. One ofname
orid
must be specified.asset-type
is the type of the asset.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting assets
- The following syntax deletes a asset by name from the specified project:
cpdctl dsjob delete-asset {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--asset-type ASSETTYPE]
project
is the name of the project that contains the asset.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the asset that is being deleted.id
is the id of the asset. One ofname
orid
must be specified.asset-type
is the type of the asset.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating attachments
- The following syntax creates an attachment to an asset in the specified project:
cpdctl dsjob create-attachment {--project PROJECT | --project-id PROJID} {--asset-name NAME | --asset-id ID} [--name ATTACHMENTNAME] --is-json --file-name FILENAME
project
is the name of the project that the asset is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.asset-name
is the name of the asset. This field must be specified.asset-id
is the id of the asset. One ofasset-name
orasset-id
must be specified.name
is the name of the attachment.is-json
specifies that the attachment file is in JSON format.file-name
is the name of the file that contains the attachment JSON. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting attachments
- The following syntax fetches an attachment by name from the specified project:
cpdctl dsjob get-attachment {--project PROJECT | --project-id PROJID} {--asset-name NAME | --asset-id ID} {--name NAME | --id ID} [--file-name FILENAME]
project
is the name of the project that contains the asset.project-id
is the id of the project. One ofproject
orproject-id
must be specified.asset-name
is the name of the asset. This field must be specified.asset-id
is the id of the asset. One ofasset-name
orasset-id
must be specified.name
is the name of the attachment.id
is the id of the attachment. One ofname
orid
must be specified.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting attachments
- The following syntax deletes a asset by name from the specified project:
cpdctl delete-attachment {--project PROJECT | --project-id PROJID} {--asset-name NAME | --asset-id ID} {--name NAME | --id ID}
project
is the name of the project that contains the asset.project-id
is the id of the project. One ofproject
orproject-id
must be specified.asset-name
is the name of the asset. This field must be specified.asset-id
is the id of the asset. One ofasset-name
orasset-id
must be specified.name
is the name of the attachment that is being deleted.id
is the id of the attachment. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing match specifications
- The following syntax displays a list of all match specifications in the specified project:
cpdctl dsjob list-match-specs {--project PROJECT | --project-id PROJID} [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the match specifications to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the match specifications in the project is displayed, one per line.sort
when specified returns the list of match specifications sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of match specifications will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the match specification id along with the name of the match specification.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting match specifications
- The following syntax fetches a match specification by name from the specified project:
cpdctl dsjob get-match-spec {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--output json|file] [--file-name FILENAME] [--with-metadata]
project
is the name of the project that contains the match specification.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the match specification.id
is the id of the match specification. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting match specifications
- The following syntax deletes a match specification by name from the specified project:
cpdctl dsjob delete-match-spec {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the match specification.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the match specification that is being deleted.id
is the id of the match specification. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Importing match specifications
- The following syntax imports a match specification into a specified project:
cpdctl dsjob import-match-spec {--project PROJECT | --project-id PROJID} [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.file-name
is the name of the file that contains the match specification.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Exporting match specifications
- The following syntax exports a match specification from a specified project to a file:
cpdctl dsjob export-match-spec {--project PROJECT | --project-id PROJID} {--name NAME | --id ID} [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the match specification.id
is the id of the match specification. One ofname
orid
must be specified.file-name
specifies the name of the .zip file to which the match specification is written. If not specified, the name or id of the match specification is the name of the file.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Copying match specifications
- The following syntax copies a match specification.
cpdctl dsjob copy-match-spec {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the match specification.id
is the id of the match specification. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing standardization rules
- The following syntax displays a list of all standardization rules in the specified project:
cpdctl dsjob list-rules {--project PROJECT | --project-id PROJID} [--sort]
project
is the name of the project that contains the standardization rules to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the standardization rules in the project is displayed, one per line.sort
when specified returns the list of standardization rules sorted in alphabetical order. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Creating standardization rules
- The following syntax creates a standardization rule in the specified project:
cpdctl dsjob create-rule {--project PROJECT | --project-id PROJID} --name NAME [--description DESCRIPTION] [--location LOCATION]
project
is the name of the project that the standardization rule is created for.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the standardization rule.description
is the description of the rule.location
is the location folder where the rule is created. Rules can only be created in the folder Customized Standardization Rules.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting standardization rules
- The following syntax fetches a standardization rule by name from the specified project:
cpdctl dsjob get-rule {--project PROJECT | --project-id PROJID} [--name NAME] [--location LOCATION] [--output json|file] [--file-name FILENAME]
project
is the name of the project that contains the standardization rule.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the standardization rule.location
is the location of the rule.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting standardization rules
- The following syntax deletes a standardization rule by name from the specified project:
cpdctl dsjob delete-rule {--project PROJECT | --project-id PROJID} [--name NAME] [--location LOCATION}
project
is the name of the project that contains the standardization rule.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the standardization rule that is being deleted.location
is the location of the rule.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Importing standardization rules
- The following syntax imports a standardization rule into a specified project:
cpdctl dsjob import-rule {--project PROJECT | --project-id PROJID} [--name NAME] [--location LOCATION] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the standardization rule.location
is the location of the rule.file-name
is the name of the file that contains the rule information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Exporting standardization rules
- The following syntax exports a standardization rule from a specified project to a file:
cpdctl dsjob export-rule {--project PROJECT | --project-id PROJID} [--name NAME] [--location LOCATION] [--file-name FILENAME]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the standardization rule.location
is the location of the rule.file-name
specifies the name of the file to which the rule is exported in .zip format.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Copying standardization rules
- The following syntax copies a standardization rule.
cpdctl dsjob copy-rule {--project PROJECT | --project-id PROJID} [--name NAME] [--location LOCATION] [--dest NAME] [--dest-location LOCATION]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the standardization rule.location
is the location of the rule.dest
is the new name of the standardization rule. This is optional and if not specified a default name will be provided.dest-location
is the location of the copied rule. If not specified the location defaults to Customized Standardization Rules.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing pipelines
- The following syntax displays a list of all pipelines in the specified project:
cpdctl dsjob list-pipelines {--project PROJECT | --project-id PROJID} [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the pipelines to list.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the pipelines in the project is displayed, one per line.sort
when specified returns the list of pipelines sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of pipelines will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.with-id
when specified prints the pipeline id along with the name of the pipeline.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Getting pipelines
- The following syntax fetches a pipeline by name from the specified project:
cpdctl dsjob get-pipeline [--project PROJECT | --project-id PROJID] [--name name | --id ID] [--output file] [--file-name <name>] [--with-metadata]
project
is the name of the project that contains the pipeline.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the pipeline.id
is the id of the pipeline. One ofname
orid
must be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.with-metadata
specifies that the output can contain metadata information.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Deleting pipelines
- The following syntax deletes a pipeline by name from the specified project:
cpdctl dsjob delete-pipeline {--project PROJECT | --project-id PROJID} {--name NAME | --id ID}
project
is the name of the project that contains the pipeline.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the pipeline that is being deleted.id
is the id of the pipeline. One ofname
orid
must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Importing pipelines
- The following syntax imports a pipeline into a specified project:
cpdctl dsjob import-pipeline [--project PROJECT | --project-id PROJID] --name name [description DESCRIPTION] [--volatile] --file-name FILENAME
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the pipeline.description
is the description of the pipeline.volatile
when specified creates a trial version of the pipeline.file-name
is the name of the file that contains the pipeline JSON. This field must be specified.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Exporting pipelines
- The following syntax exports a pipeline from a specified project to a file:
cpdctl dsjob export-pipeline [--project PROJECT | --project-id PROJID] [--name name | --id ID] [--volatile] [--format TEMPLATE|FLOW|ALL] [--output file] [--file-name <name>]
project
is the name of the project.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the pipeline.id
is the id of the pipeline. One ofname
orid
must be specified.volatile
when specified creates a trial version of the pipeline.format
specifies whether to export the pipeline template, pipeline flow, or both.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the exported pipeline JSON is written to.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing pipeline versions
- The following syntax displays a list of all pipeline versions in the specified project:
cpdctl dsjob list-pipeline-versions {--project PROJECT | --project-id PROJID} [--sort | --sort-by-time] [--with-id]
project
is the name of the project that contains the pipeline.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the pipeline versions in the project is displayed, one per line.name
is the name of the pipeline.id
is the id of the pipeline. One ofname
orid
must be specified.sort
when specified returns the list of pipeline versions sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of pipeline versions will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Listing pipeline runs
- The following syntax displays a list of all pipeline runs in the specified project:
cpdctl dsjob list-pipeline-runs [--project PROJECT | --project-id PROJID] [--name name | --id ID] [--sort | --sort-by-time] [--detail] [--output file|json] [--file-name FILENAME]
project
is the name of the project that contains the pipeline.project-id
is the id of the project. One ofproject
orproject-id
must be specified. A list of all the pipeline runs in the project is displayed, one per line.name
is the name of the pipeline.id
is the id of the pipeline. One ofname
orid
must be specified.sort
when specified returns the list of pipeline runs sorted in alphabetical order. This field is optional.sort-by-time
when specified the list of pipeline runs will be sorted by time of creation with latest at the top of the list. One ofsort
orsort-by-time
can be specified.detail
when specified prints the pipeline run details.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Running pipelines
-
A pipeline run is triggered by creating a job for the pipeline and running it. The following syntax runs a pipeline in the specified project:
cpdctl dsjob run-pipeline [--project PROJECT | --project-id PROJID] [--name name | --id ID] [--job-name name] [--description description] [--version VERSION] [--param PARAM] [--param-file FILENAME] [--env ENVJSON] [--wait SEC]
project
is the name of the project that contains the pipeline.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the pipeline.id
is the id of the pipeline. One ofname
orid
must be specified.job-name
is the name of the job to be created or used. This field is optional.description
is the description of the job that is run.mode
can be specified to start, cancel, or restart a job. The value of mode defaults to NORMAL, which starts a job run. If the value ofmode
is RESET, a running job is canceled. RESTART restarts a job run. This field is optional.version
specifies the version of the pipeline that is run.param
specifies a parameter value to pass to the job. The value is in the formatname=value
, where name is the parameter name and value is the value to be set. This flag can be repeated, ex:--param k1=v1 --param k2=v2
paramfile
specifies a file that contains the parameter values to pass to the job. This field is not implemented currently.env
specifies the environment in which job is run.env
is specified as a key=value pair. Key `env` or `env-id` can be used to chose a runtime environment. Example:--env $APT_DUMP_SCORE=true --env env=ds-px-default
wait
the job run waits for the specified amount of time for the job to finish. The job logs are printed to the output until the job is completed or the wait time expires. The return status indicates whether the job has finished, finished with warning, raised an error, or timed out after waiting. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.
- Printing pipeline run logs
- The following syntax fetches run logs of a pipeline run in the specified project:
cpdctl dsjob get-pipeline-logs [--project PROJECT | --project-id PROJID] [--name name | --id ID] [--run-id RUNID] [--output file] [--file-name <name>]
project
is the name of the project that contains the pipeline.project-id
is the id of the project. One ofproject
orproject-id
must be specified.name
is the name of the pipeline.id
is the id of the pipeline. One ofname
orid
must be specified.run-id
if specified, the logs for that run id is printed. If not specified, the logs from the latest run are printed.output
specifies the format of the output. This field is optional.file-name
specifies the name of the file to which the output is written. This field is optional.
A status code is printed to the output. A status code of 0 indicates successful completion of the command.