What's new and changed?

System configurations and integrations

Inclusive terminology

While IBM values the use of inclusive language, terms that are outside of IBM's direct influence are sometimes required for the sake of maintaining user understanding. As other industry leaders join IBM in embracing the use of inclusive language, IBM will continue to update the documentation to reflect those changes.

Installing, upgrading and configuring

Verify the integrity and validate the authenticity of your downloaded installation packages from IBM Passport Advantage®

Now, for added security, after you have downloaded your IBM Spectrum Conductor installation packages from IBM Passport Advantage, you can verify the integrity and validate the authenticity of these files. Simply run a single jarsigner utility. For details on using jarsigner, see Downloading entitled packages.

Set the built-in Admin user password during fresh installations

For added security, you can now set the password for the built-in Admin user, rather than use a preset password. Complete this password setting during a fresh installation of IBM Spectrum Conductor on your primary host using the new egoconfig setpassword command. When you run the new egoconfig join command after installation and before setting product entitlement, the system reminds you to run the new egoconfig setpassword command.

New parameter to handle authentication requests in separate threads in VEMKD process

With the new EGO_AUTH_THREAD_NUMBER parameter, you can specify the number of threads that are needed to handle authentication requests in the VEMKD process. For more information, see ego.conf reference.

Instance groups and notebooks

Support high availability for the Dask manager

To enable high availability for the Dask manager, specify the High availability recovery directory, where the Dask manager writes recovery data that is accessible by all Dask manager candidates. For more information, see the Configuring Dask settings for an instance group topic.

New Jupyter notebook environment variables

This version of IBM Spectrum Conductor includes several new Jupyter environment variables:

Environment variable name	Purpose
JUPYTER_SERVICE_ACTIVITY_TIMEOUT	Use the new JUPYTER_SERVICE_ACTIVITY_TIMEOUT Jupyter environment variable to specify the number of seconds to pass before the server shuts down when the server is idle.
NB_EXTRA_CONF_PARAM_parameterName	Use the new NB_EXTRA_CONF_PARAM_parameterName and JEG_EXTRA_CONF_PARAM_parameterName environment variables to set extra notebook and Jupyter enterprise gateway configuration parameters.
REMOVE_CONFIG_DIR_ON_START	Use the new REMOVE_CONFIG_DIR_ON_START Jupyter environment variable to specify how the Jupyter start script handles notebook configuration directory each time a Jupyter notebook starts.

For more information about these new Jupyter environment variables, see Jupyter notebook environment variables.

New data connectors for instance groups to use data on remote file systems

Configure the IBM Cloud Object Storage and Amazon S3 data connectors for instance groups to switch between data sources on demand. IBM Spectrum Conductor can now be configured to access Amazon S3 file systems by using Amazon S3 data connectors. For more information, see Data connectors.

New operations to streamline the process of moving data connectors between clusters

You can remove, export, and import data connectors to move from one cluster to the other:

• For more information on how to remove existing data connectors that are no longer needed from the cluster, see Removing data connectors.
• For more information on how to export an existing data connector from the cluster management console to the file system, see Exporting data connectors.
• For more information on how to import an external data connector package that was exported from the cluster, see Importing data connectors.

New operations to manage instance level configurations for Anaconda or Miniconda distributions

• Fetch configurations for Anaconda or Miniconda distribution instances. For more information, see Retrieving Anaconda configurations.
• Add new configuration parameter values, remove, or update existing configuration parameter values for Anaconda or Miniconda distribution instances. For more information, see Managing Anaconda configurations.

Override configuration settings for SparkCleanup service

Use registration or modification services to override the settings set globally during SparkCleanup profile setup. For more information, see Configuring cleanup for instance groups during registration or modification.

New environment variable for the SparkCleanup service

Use the new CONDUCTOR_CLEANUP_DEPLOY_PACKAGES environment variable to specify whether the SparkCleanup service cleans up SOAM deployment packages. For more information, see Configuring cleanup for instance groups during profile setup topic.

When you set this CONDUCTOR_CLEANUP_DEPLOY_PACKAGES environment variable to false, you can configure the egodeploy service to use the existing SOAM deployment files as opposed to downloading the files again. For more information, see Disabling deployment package cleanup by redepolying existing packages topic.

Spark applications

New parameter to automatically restart driver during disruptions

With the new spark.ego.driver.blockhost.supervise parameter, you can automatically restart the driver if the driver fails to start with a specific range of driver exit codes. For more information, see Spark on EGO Spark application parameters.

Cluster and resource management

Extended RESTful APIs for multidimensional resource scheduling

In addition to the existing GET method to create a new multidimensional resource plan, this release of IBM Spectrum Conductor includes eight new RESTful APIs for managing multidimensional resource scheduling. See the MultidimensionalScheduling section within the curent version of the REST API for /platform/rest/ego/v1 for details; the following table highlights each API:

Table 1. RESTful APIs for multidimensional resource scheduling

Method	Description and URL	New or existing
GET	Get a list of all multidimensional resource plans in the cluster https://host_name:port/platform/rest/ego/v1/mdsplans	Existing
GET	Get a multidimensional resource plan by name https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}	New
POST	Create a new multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans	New
PUT	Modify an existing multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}	New
DELETE	Delete a multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}	New
GET	Export the XML of all existing multidimensional resource plans in the cluster https://host_name:port/platform/rest/ego/v1/mdsplans/xml	New
GET	Export the XML for a single existing multidimensional resource plan in the cluster https://host_name:port/platform/rest/ego/v1/mdsplans/xml/{mdsPlanName}	New
PUT	Import the XML of a multidimensional resource plan to update an existing multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/{mdsPlanName}	New
POST	Import the XML of a multidimensional resource plan to create a new multidimensional resource plan https://host_name:port/platform/rest/ego/v1/mdsplans/xml	New

Set overcommit ratios for resource metrics at the multidimensional resource group level

In addition to setting resource overcommit ratios at the multidimensional resource plan level, you can now set this for each resource group within your plan. Leverage this enhancement following the steps in Setting resource overcommit ratios in your multidimensional resource plan.

New parameter to set how LIM gets the IP address from a hostname

Configure the new EGO_GET_HOST_IP_ADDRESS_WITHOUT_LIM_DNS_CACHE parameter in the ego.conf file to Y, to specify for LIM to ignore the host's DNS cache and to use the EGO host file or system API instead. This configuration is useful if unavailable hosts cannot be removed automatically.

Performance and stability

Configure the number of connections that can remain in the LIM and VEMKD backlogs

For improved connection stability, you can now configure the number of connections that can remain in the LIM and VEMKD backlogs, by configuring the new EGO_PLATCOMMDRV_BACKLOG_SIZE_ON_LINUX setting in the ego.conf configuration file.

Security

Terminology update: TLS (Transport Layer Security) protocol for security

Starting in IBM Spectrum Conductor 2.5.1, the documentation has been updated to refer SSL as TLS for the IBM Spectrum Conductor security protocol; functionality of IBM Spectrum Conductor security has not changed. TLS uses encryption to protect the transfer of data and information, in the same way as SSL.

This terminology change is simply to keep up with the evolving industry terminology change. IBM Spectrum Conductor documentation now refers to TLS as the supported protocol, as much as possible. Some parameter names, environment variables, attribute values, messages, and so on, continue to SSL in their names, but can be treated as TLS protocol.

Secure built-in Admin and Guest user credentials

For added security, the Admin and Guest user credential values and usage have changed:

Admin

You can now set the password for the built-in Admin user, rather than use a preset password by following Set the built-in Admin user password during fresh installations.

Guest

You no longer need to use Guest as the username and password. For anywhere Guest is used (such as a sample or an example), treat as for explanatory purposes only; replace Guest with your real username and password for your cluster.

Host factory framework for cloud bursting

Supporting IBM Cloud Virtual Servers for VPC Gen 2 as a cloud service provider

Host factory for IBM Spectrum Conductor adds IBM Cloud Gen 2 as an additional cloud service provider. For more information, see Configuring your cluster for IBM Cloud Gen 2, ibmcloudgen2prov_config.json, and ibmcloudgen2prov_templates.json topics.

Manage requestor plug-ins and instances, or provider plug-ins and instances

You can view, create, update, and delete a requestor plug-in, requestor instance, provider plug-in, or provider instance from the cluster management console:

• Viewing requestor plug-ins and instances
• Viewing provider plug-ins and instances

Alternatively, you can use RESTful APIs to add, update, delete, and get a list for requestor plug-in, requestor instance, provider plug-in, or provider instance:

Table 2. Host factory REST APIs for requestor plug-in

Method	Description and URL
POST	Create a requestor plug-in https://host_name:port/platform/rest/hostfactory/requestorplugins
GET	Get a list of requestor plug-ins https://host_name:port/platform/rest/hostfactory/requestorplugins/$plugin_name
PUT	Update a requestor plug-in https://host_name:port/platform/rest/hostfactory/requestorplugins/$plugin_name
DELETE	Remove a requestor plug-in https://host_name:port/platform/rest/hostfactory/requestorplugins/$plugin_name

Table 3. Host factory REST APIs for provider plug-in

Method	Description and URL
POST	Create a provider plug-in https://host_name:port/platform/rest/hostfactory/providerplugins
GET	Get a list of provider plug-ins https://host_name:port/platform/rest/hostfactory/providerplugins/$plugin_name
PUT	Update a provider plug-in https://host_name:port/platform/rest/hostfactory/providerplugins/$plugin_name
DELETE	Remove a provider plug-in https://host_name:port/platform/rest/hostfactory/providerplugins/$plugin_name

Table 4. Host factory REST APIs for requestor instance

Method	Description and URL
POST	Create a requestor instance https://host_name:port/platform/rest/hostfactory/requestors
GET	Get a list of requestor instance https://host_name:port/platform/rest/hostfactory/requestors/$instance_name
PUT	Update a requestor instance https://host_name:port/platform/rest/hostfactory/requestors/$instance_name
DELETE	Remove a requestor instance https://host_name:port/platform/rest/hostfactory/requestors/$instance_name

Table 5. Host factory REST APIs for provider instance

Method	Description and URL
POST	Create a provider instance https://host_name:port/platform/rest/hostfactory/providers
GET	Get a list of provider instance https://host_name:port/platform/rest/hostfactory/providers/$instance_name
PUT	Update a provider instance https://host_name:port/platform/rest/hostfactory/providers/$instance_name
DELETE	Remove a provider instance https://host_name:port/platform/rest/hostfactory/providers/$instance_name

View and update configurations

View and update service-level host factory configuration. For more information, see Viewing configurations.

Enhancements for the AWS provider plug-in

New parameters for timeout and retries

Introducing three new parameters that you can use to get status of instances from the AWS provider. Configure AWS_INSTANCE_PENDING_TIMEOUT_SEC for specifying the timeout before a pending instance is terminated, AWS_DESCRIBE_REQUEST_RETRY_ATTEMPTS for the number of retries, and AWS_DESCRIBE_REQUEST_INTERVAL for delay between retries. For more launch information about how to configure these new parameters, see awsprov_config.json reference.

New parameters for AWS EC2 instances

Introducing six new parameters for AWS EC2 instances:

• Configure volumeType and iops for EC2 On-Demand and Spot instances.
• Configure percentOnDemand, vmTypesOnDemand, vmTypesPriority, or allocationStrategyOnDemand for EC2 On-Demand instances.

For more details on how to configure these new parameters, see awsprov_templates.json reference.

New option for allocationStrategy

Introducing a new option for spot instance allocationStrategy parameter. Specify the new capacityOptimizedPrioritized option if you want to give certain Spot instances types a higher chance of launching first. For more information about the new capacityOptimizedPrioritized option for the allocationStrategy parameter, see awsprov_templates.json reference.

Removed ncores as a value for the attributes parameter in all host template provisioning files

The attributes parameter has been changed for the host template provisioning files: the ncores value is no longer supported; use the ncpus value instead. All host template files (provider_nameprov_templates.json) under Reference for host factory and template examples have been updated to no longer show ncores in the templates.

For backwards compatibility, the HostFactory service and plug-ins continue to work if ncores was specified previously. IBM Spectrum Conductor uses the default value of 1 (that is, if you configured ncores to another value, it will not take effect and the value of 1 will be used instead).