Job configuration overlays reference
You can specify or change configuration parameters when you use the submitjob command or the previewsubmitjob command. The settings control submission constraints, host constraints and locations, and operator status. Parameters are captured and updated in the configuration JSON file.
Submit job parameters format
The following categories classify parameters as input or output related:
Attribute | Type | Value |
---|---|---|
comment (optional) | String | Comment text. |
jobConfigOverlays (optional) | Array JobConfigOverlay | Each element of the array contains the job configuration settings. Only the first jobConfigOverlay element is currently supported. Other elements in the array are ignored. |
results (optional) | Results | Produced from the previewsubmitjob command. This parameter is output only and is ignored for job preview or submission. Results can be successful, partially successful, or failure. |
- jobConfigOverlays
- Single element array of jobConfigOverlays.
- Results
Array of strings that is produced from the previewsubmitjob command. This parameter is output only and is ignored for job preview or submission. Each element of the array describes a resulting PE.
JobConfigOverlay
The following parameter defines the contents of the job configuration overlay:
Attribute | Type | Value |
---|---|---|
jobConfig (optional) | Object JobConfig | Job configuration values |
deploymentConfig | Object DeploymentConfig | Deployment configuration values. |
operatorConfigs (optional) (ignored when fusionScheme=legacy) |
ObjectOperatorConfigs | Operator configuration values. |
configInstructions (optional) | ObjectConfigInstructions | Configuration instructions. |
JobConfig
The following parameter defines the contents of the job configuration:
Attribute | Type | Value |
---|---|---|
jobName (optional) | String | The name that is assigned to the job. |
jobGroup (optional) | String | The job group to use to control permissions for the submitted job. |
submissionParameters (optional) | ArraySubmissionParameters | Job-specific key: value submission parameters. |
dataDirectory (optional) | String | Specifies the location of the data directory. |
preloadApplicationBundles (optional) | Boolean | Specifies whether to preload the job onto all resources in the instance, even if the job is not currently needed on each. The valid values are true and false; false is the default value. Preloading the job can improve the performance if the PEs are relocated to a new resource. |
tracing (optional) | String | Specifies the trace setting for the PEs. The valid values are error, warn, info, debug, and trace. |
SubmissionParameter
The following elements comprise the submission parameters:
Attribute | Type | Value |
---|---|---|
name | String | Parameter name. |
value | String | Parameter value. |
DeploymentConfig
The following elements comprise the contents of the deployment configuration:
Attribute | Type | Value |
---|---|---|
fusionScheme (optional) | String |
|
fusionTargetPeCount (optional) (requires fusionScheme=manual orinstance.fusionScheme=manual) |
Number | Use this option to set the number of PEs to create. fusionScheme must be set to manual. |
placementScheme | String |
For both balanced schemes, the balancing pass selects the candidate host that has the lowest PE to core ratio. If there is a tie, the balancing process chooses the host with the lowest ldavg, normalized to number of cores. |
threadingModel | String |
|
dynamicThreadingThreadCount (optional) | Number | Specifies the number of threads that scheduled ports should use. Inherits its value from instance.dynamicThreadingThreadCount, if set. Valid values are integers greater than or equal to 1. This setting can only be used if the configuration threadingModel is set to dynamic. If dynamicThreadingElastic is set to true, the number of threads might change at runtime. |
dynamicThreadingElastic (optional) | Boolean | Specifies if elasticity is on or off for dynamically threaded ports. Valid values are true for on, and false for off. If elasticity is on, the number of dynamic threads can change at runtime to maximize throughput. This setting can only be used if the application configuration threadingModel is set to dynamic. |
parallelRegionConfig (optional) (ignored if fusionScheme=legacy) |
Object parallelRegionConfig | Configuration information for all the ParallelRegions in the job. |
parallelRegionConfig
The following elements comprise the submission parameters:
Attribute | Type | Value |
---|---|---|
fusionType | String |
|
OperatorConfigs
The following parameters define the contents of the operator configurations:
Attribute | Type | Value |
---|---|---|
operators (optional) | Array Operator | Configuration attributes for individual operators. Not all existing operators must be in the array, but only operators that are defined by the application are allowed, and they can exist only one time in this array. |
hostColocationGroups (optional) | Array LocationGroup | Each element of the array represents a hostColocationGroup, where each member of the group must be placed onto the same host as other operators in this group. These groups are combined with groups that are defined in the application bundle. |
hostExlocationGroups (optional) | ArrayLocationGroup | Each element of the array represents a hostExlocationGroup, where each member of the group cannot be placed onto the same host as any other operators in this group. These groups are combined with groups that are defined in the application bundle. |
partitionColocationGroups (optional) | ArrayLocationGroup | Each element of the array represents a partitionColocationGroup, where each member of the group must be fused together with other operators in the group into the same PE. These groups are combined with groups that are defined in the application bundle. |
partitionExlocationGroups (optional) | ArrayLocationGroup | Each element of the array represents a partitionExlocationGroup, where each member of this group cannot be fused together with any other operators from this group into the same PE. These groups are combined with groups that are defined in the application bundle. |
poolPlacementGroups (optional) | Array PoolPlacementGroup | Each element of the array represents a hostpool definition. |
LocationGroup
The following parameters define the location groups:
Attribute | Type | Value |
---|---|---|
groupName (optional) | String | Name of a location group. If this name matches the name of a group that is defined in the application bundle, this group replaces the group in the bundle. |
members (optional) | Array String | Each element of the array is a nameSpec of the operator or operators that are included in this group. If this attribute is not specified, or the array is empty, then a group with a matching groupName that is defined in the application bundle is removed. |
PoolPlacementGroup
The following parameters define the details for pool placement groups:
Attribute | Type | Value |
---|---|---|
name (optional) | String | Name of the hostpool. |
operatorsInPool (optional) | Array OperatorInPool | A specific operator can be specified in only one PoolPlacementGroup or in the declaredHost attribute of the Operator. |
membershipMode (optional) | String | SHARED: Hosts that are members of this pool are allowed to be
in other pools as well. This setting is the default. EXCLUSIVE: Hosts that are members of this pool are not allowed to be in any other pools, whether this job or any other jobs. The value for size must be specified. |
size (optional) | Number | Optional number. Defines the number of hosts that are allowed in the host
pool. If specified, the number must be greater than 0. If declaredHosts size is greater than 0, then the value that is specified for size is ignored. |
tags (optional) (exclusive with declaredHosts) | ArrayString | Defines what the tagging requirements are for hosts to be considered for membership of this pool. |
declaredHosts (optional) (exclusive with tags) (exclusive with membershipMode=Exclusive) | ArrayString | Defined hosts that are members of this host pool. |
OperatorInPool
The following parameters define the operator pools:
Attribute | Type | Value |
---|---|---|
nameSpec | String | nameSpec of the operators in the pool. |
inPoolIndex (optional) | Number | Optional number. The index within the host pool, for example, the fourth host listed in the host pool. This value can be specified only as part of a PoolPlacementGroup that has a size specified. The index number must be smaller than the specified size. |
Operator
The following parameters define the details for operator configuration:
Name | Type | Value |
---|---|---|
nameSpec | String | Specification of a name of an existing operator instance in the application
after UDP transformations. The name might contain zero or more '*' characters in the specification.
The '*' represents 0 or more occurrences of any legal name character. For example, "Main.*.MyComp2.Filter[*]" '[ ]' are normal characters, and do not have any regular expression meaning. |
restartable (optional) | Boolean | true: A PE that contains this operator is able to be
restarted. (default) false: A PE that contains this operator is not able to be restarted. |
relocatable (optional) | Boolean | true: A PE that contains this operator is able to be
relocated. (default) false: A PE that contains this operator is not able to be relocated. |
partitionIsolation (optional) | Boolean | true: A PE that contains this operator cannot be fused with
any other operator. false: A PE that contains this operator can be fused with other operators. (default) |
hostIsolation (optional) | Boolean | true: A PE that contains this operator cannot be placed on
the same host as any other PE. false: A PE that contains this operator can be placed on the same host with other PEs. (default) |
declaredHost (optional) | String | A PE that contains this operator must be placed on this specific host. If this attribute is specified, then the operator cannot be included in any of the PoolPlacement sections. |
ConfigInstructions
The following parameters define the details for configuration instructions
Name | Type | Value |
---|---|---|
ignoreBundleRelationalPlacements(optional) (ignored when fusionScheme=legacy) |
String | PARTITION: Clear all partition relational placements from
bundle (partitionIsolation, partitionColocationGroups,
partitionExlocationGroups). HOST: Clear all host relational placements from bundle (hostIsolation, hostColocationGroups, hostExlocationGroups). ALL: Clear partition and host relational placements. NONE: Bundle relational placements are left as is (default). Tip: Be careful when you use this keyword because the effects can counteract the
intentions of the application developer.
|
ignoreBundleHostPlacements
(optional) (ignored when fusionScheme=legacy) |
Boolean | true: All operators are placed in the default pool. (default=false) Tip: Be careful when you use this keyword because the effects can counteract the
intentions of the application developer.
|
convertTagSet (optional) | Array tagConversion | Replace tags in the poolPlacementGroups with tags in the
convertTagSet. If poolPlacementGroup.tags is a superset of originalTagSet, then tags in originalTagSet are removed from poolPlacementGroup.tags, and tags in targetTagSet are inserted in its place. If the originalTagSet is not specified, then a poolPlacementGroup with an empty tag set, known as a default hostpool, has its tags changed to include targetTagSet. The parameter does not convert tags within the jobConfigOverlay. |
changeThreadedPortSettings (optional) | Object ThreadedPortSetting | Change settings for all threaded ports that are defined in the application. |
ThreadedPortSetting
The following parameter defines the threaded port settings:
Attribute | Type | Value |
---|---|---|
queueSize | Number | Size of threaded port queue. Must be greater than or equal to 1. |
TagConversion
The following parameters define the tag conversion:
Attribute | Type | Value |
---|---|---|
originalTagSet | Array String | An array of strings that represent a list of tags to convert from. |
targetTagSet | Array String | An array of strings that represent a list of tags to convert to. |
OriginalTagSet | TargetTagSet | poolPlacementGroupTags Before | poolPlacementGroupTags After | Notes |
---|---|---|---|---|
"A" | "X" | "A,B" | "X,B" | One to one |
"A" | "X,Y" | "A,B" | "X,Y,B" | One to many |
"A,B" | "Y" | "A,B,C" | "Y,C" | Many to one |
"A,B" | "W,Y,Z" | "A,B,C" | "W,Y,Z,C" | Many to many |
"A,B" | "Z" | "A,C,D" Not a superset; missing B. |
"A,C,D" | No change, since not all tags exist in the "Before" set |
" " | "X,Y" | " " | "X,Y" | Adding tags to default hostpool. |
Results
The following parameters define the submission results. These parameters are output only and are ignored on input:
Attribute | Type | Value |
---|---|---|
placementResults (optional) | Array PlacementResult | Each element of the array describes a resulting PE. This field is not specified if fusionErrors occur, but is specified if only placementErrors occur. |
fusionErrors (optional) | Array FusionError | Each element of the array provides information on errors that occurred while the system is trying to fuse operators into a PE. |
placementErrors (optional) | Array PlacementError | Each element of the array provides information on errors that occurred while the system is trying to place a PE onto a host. |
PlacementResult
If the fusion phase completes successfully, a placementResults element is returned whether the placementPhase was successful or not. If the placement phase fails, then the hostPlacement field is not set. This partial return is useful if you want to use previewsubmitjob get feedback on the fusing phase before you have your test instance fully configured.
Attribute | Type | Value |
---|---|---|
peId (optional) | Number | PE identifier. |
names | Array String | Names of the operators to fuse together into this PE. |
tags (optional) (exclusive with declaredHosts) | Array String | The host placement tags that are associated with this fused PE. |
declaredHosts (optional) (exclusive with tags) | Array String | Any declared host that this PE is associated with. |
hostPlacement (optional) | String | The target host for PE placement if the job was submitted. If
placementPhase is not successful, then this field is not set. Since the current runtime situation is temporal and affects the host selection, this hostPlacement might not be the same for subsequent submitjob operations. |
FusionError
The FusionError information describes the set of operators that were intended to be fused together, as well as what factors might have prevented successful fusion.
Attribute | Type | Value |
---|---|---|
operatorNames | Array String | Names of operators that were not successfully fused. |
detailedProblemMessages | Array Message | Messages that describe in detail the problems that were detected that
prevented successful fusion. The messages are scoped to the collection of operators and not to a
single operator. For example: CDISR3203E One or more of the operators are specified as partitionIsolate which prohibits from being fused with any other operator. CDISR3210E Different indexes [0, 1] within the 0 hostpool were specified by the operators. |
PlacementError
The PE placement process is hierarchical in nature. A job contains one or more PE colocation groups. A PE colocation group contains one or more PEs. A PE contains one or more operators. The placement process cycles through various constraints to arrive at a candidate host set for each placementNode. If the host set is empty, that node in the placement tree cannot be placed.
The error messages are also hierarchical. In the previously described scenario, the reported error is not the last constraint that makes the set empty, but rather a composite message structure that describes the bigger picture of what causes the failure. At each node in the placement tree, there can be constraints that nominate or eliminate hosts to the candidate host set.
For example, each child node of a node nominates its candidate host set to its parent's candidate set. The parent node eliminates all hosts from its candidate set that are not included in each of its child's candidate set. That is, the parent set starts with the intersection of all child sets.
Attribute | Type | Value |
---|---|---|
placementNodeInfo | PlacementNodeInfo | Contains information about the overall status of the placement node, including whether the node is able to be successfully placed. |
contributingMessages | Array Message | A set of messages that describe the contributions that produce the resulting
placementNodeInfo status. Some contributing messages describe why a set of hosts is nominated to the
candidate set, for example: CDISR3051I The following hosts were included in the scheduling process because the hosts are eligible to be included in the MyHostPool host pool by having the following host pool tags. The hosts are: 10.4.24.226, 10.4.40.236. The host pool tags that are shared by the hosts are: [Red,Blue]. Some contributing messages describe why a set of hosts is eliminated from the candidate set, for example: CDISR3066I: Because the following hosts are not supported by all the operators in the partition, these hosts cannot be included in the scheduling process: 10.4.24.226, 10.4.40.247. |
childContributions | Array PlacementError | A set of placement errors that detail how the child placement nodes contribute to these placement node results. |
PlacementNodeInfo
Attribute | Type | Value |
---|---|---|
success | Boolean | If the value is true, this node has at least one valid
candidate host. If the value is false, there are no candidate hosts remaining for this placementNode, and placement fails for this node. |
name (optional) (exclusive with peOperatorSet and colocatedPeSet) | String | If present, the placement node is related to the operator that is named by this parameter. |
peId (optional) (exclusive with name and colocatedPeSet | String | If present, the placement node is related to a PE. The peOperatorSet field contains the related operator name set for this PE. |
peOperatorSet (optional) (exclusive with name and colocatedPeSet) | Array String | If present, the placement node is related to a PE, and this is the set of operator names that make up the PE. The ID for this PE is specified in the peID field. |
colocatedPeSet (optional) (exclusive with name and peOperatorSet) | Array String | If present, this placement node is related to a colocated set of PEs. This parameter expresses the array of PE IDs as a string. |
placementMessage | Message | A message that describes the placement status for this placement node. For
example: CDISR3046E The processing element ID 3: op1 cannot be scheduled. For more information, see the messages that follow this one. CDISR3047I: A constraint on the op1 operator requires that it be scheduled from the following hosts: 10.4.24.226, 10.4.40.236.. For more information, see the messages that follow this one. |
Message
Attribute | Type | Value |
---|---|---|
messageCode | String | The code for the message, for example, CDISR3200E . |
substitutionTexts ) | Array String | The substitution text string for the message. |
displayMessage (optional) | String | Human readable display message with substitutionTexts inserted and
instantiated by the client in the client locale. This field is transient and is included to improve the readability of error messages in the JSON file. The field is optional for the previewsubmitjob client. |