Job configuration overlays
A job configuration overlay file is a JSON file that contains name-value pairs for the submission-time configuration parameters that you want to use. A job configuration overlay file enables you to define, save, and distribute the submission-time configuration.
A job configuration overlay file is one way that you can specify submission-time configuration parameters for a job before you deploy a stream processing application to your environment. (Alternatively, you can specify submission-time configuration parameters from the IBM® Streams Console, from the Streams Studio, or by using the streamtool submitjob command.)
However, unlike manually specifying submission-time configuration parameters when you submit a job, a job configuration overlay file enables you to easily reuse or share a configuration. It also reduces the likelihood of entering information incorrectly if you want to specify submission-time configuration parameters from the interactive streamtool interface.
You can manually create a job configuration overlay file, or you can generate a job configuration overlay file when you preview how a job will be submitted to your instance. For more information about generating a job configuration overlay file from a preview, see Previewing a job before you deploy it.
For more information about how you can use submission-time configuration parameters to improve the run-time performance of the application without recompiling it, see Configuring submission-time parameters to optimize job deployment.
Structure of a job configuration overlay file
The following outline shows the order of the sections in a job configuration overlay file:
comment
jobConfigOverlays
jobConfig
jobName
jobGroup
submissionParameters
dataDirectory
preloadApplicationBundles
tracing
deploymentConfig
fusionScheme
fusionTargetPeCount
placementScheme
threadingModel
dynamicThreadingThreadCount
dynamicThreadingElastic
parallelRegionConfig
fusionType
operatorConfigs
operators
nameSpec
restartable
relocatable
partitionIsolation
hostIsolation
declaredHost
hostColocationGroups
groupName
members
hostExlocationGroups
groupName
members
partitionColocationGroups
groupName
members
partitionExplocationGroups
groupName
members
poolPlacementGroups
name
OperatorsInPool
nameSpec
inPoolIndex
membershipMode
size
tags
declaredHosts
configInstructions
ignoreBundleRelationalPlacements
ignoreBundleHostPlacements
convertTagSet
originalTagSet
targetTagSet
changeThreadedPortSettings
queueSize
For more information about the parameters that you can specify in a job configuration overlay file, see Job configuration overlay parameters
The job configuration overlay file that is generated when you run a preview includes all of the settings that were configured for the job. For complex applications, the job configuration overlay file might be very long and contain a large number of entries.
If you want to modify a small number of settings, you can create a job configuration overlay file that includes only the relevant sections and parameters. Only the settings that are specified in the job configuration overlay file are changed.
Syntax of a job configuration overlay file
If you create or edit a job configuration overlay file, you must ensure that you use the correct syntax. Specifically, ensure that you include the commas in the correct locations and that you include opening and closing braces, brackets, and double quotation marks.
The following code demonstrates the correct syntax for a job configuration overlay file:
{
"jobConfigOverlays": [
{
"jobConfig": {
"jobName": "myJob",
"jobGroup": "myJobGroup",
"dataDirectory": "myDataDir",
"submissionParameters": [
{ "name": "streams::pe.display", "value": "host_name:port_number" }
]
},
"deploymentConfig": {
"placementScheme": "balancedInstance",
"fusionScheme": "automatic",
},
"operatorConfigs": {
"operators": [
{ "nameSpec": "Op_1", "declaredHost": "hostB" },
{ "nameSpec": "Op_2", "restartable": "true" },
{ "nameSpec": "Op_3", "hostIsolation": "true" },
],
"poolPlacementGroups": [
{
"name": "myPool".
"size": 10,
"tags": [ "tag1", "tag2" ]
"operatorsInPool": [
{ "nameSpec": "Op_2" },
{ "nameSpec": "Op_3" },
]
}
]
}
}
]
}
Using a job configuration overlay file
You can use a job configuration overlay file when you preview a job or when you submit a job:- From IBM Streams Studio, specify the job configuration overlay file from the SPL Application tab when you run or preview a job.
- From Streams Console, import the job configuration overlay file when you run or preview a job. You can import it by clicking the Import icon at the top of the page and selecting Import job configuration file.
- From interactive streamtool interface, run the appropriate command:
- To preview a job with a job configuration overlay file, run the streamtool
previewsubmitjob command and specify the name of your job configuration overlay
file:
streamtool previewsubmitjob -g jobConfigOverlay.json myAppBundle.sab
- To run a job with a job configuration overlay file, run the streamtool
submitjob command and specify the name of your job configuration overlay
file:
streamtool submitjob -g jobConfigOverlay.json myAppBundle.sab
- To preview a job with a job configuration overlay file, run the streamtool
previewsubmitjob command and specify the name of your job configuration overlay
file: