Enabling multiple conductor (PX runtime) pods in DataStage

Important: IBM Cloud Pak® for Data Version 4.7 will reach end of support (EOS) on 31 July, 2025. For more information, see the Discontinuance of service announcement for IBM Cloud Pak for Data Version 4.X.

Upgrade to IBM Software Hub Version 5.1 before IBM Cloud Pak for Data Version 4.7 reaches end of support. For more information, see Upgrading IBM Software Hub in the IBM Software Hub Version 5.1 documentation.

You can increase compute availability and improve workload balancing by increasing the number of DataStage® PX runtime conductor pods.

About this task

Compute pods can scale horizontally and vertically, with dynamic workload management distributing the workload across the available compute pods. By default, runtime instances are created with one DataStage PX runtime conductor pod. Each job is executed by one conductor pod across multiple compute pods. You can increase compute availability by increasing the number of PX runtime conductor pods. New jobs can be submitted as long as a conductor pod is available. Multiple pods from the same service will avoid deploying on the same worker node.

Procedure

Set the number of replicas for px-runtime (DataStage PX instance conductor) to 2 with the following command. You can scale the replicas to 2 or more.
```
oc scale deploy <px-runtime-deployment> --replicas=2
```
Example:
```
oc scale deploy px-instance-ibm-datastage-px-runtime --replicas=2
```

Modify the DataStage PX runtime instance custom resource (CR) to update the default. Otherwise, the next update to the CR will revert the change.

# retrieve pxruntime cr
oc get pxruntime

# edit pxruntime cr
oc edit pxruntime <cr-name>

spec:
  ignoreForMaintenance: false  # set to true to ignore refresh
  scale:                                        # override the t-shirt size
    pxcompute:           
      replicas: 2
      resources:
        limits:
          cpu: 3
          memory: 8Gi
        requests:
          cpu: 1
          memory: 2Gi
    pxruntime:
      replicas: 2
      resources:
        limits:
          cpu: 2
          memory: 4Gi
        requests:
          cpu: 500m
          memory: 2Gi